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Abstract 


This study examined teaeher pereeptions of online and faee-to-faee professional 
development (PD) in four large sehool distriets implementing an integrated sehool improvement 
initiative. The online PD eomponent of this study was designed to support eore 9th grade 
mathematies eurrieulum, and aligned safety net programs in literaey and mathematies. Study 
groups, a training format that provides materials to faeilitate faee-to-faee teaeher team meetings, 
supported eollaboration within teaeher teams in middle and high sehools. Survey and interview 
data on 101 individuals — ineluding 51 teaehers who partieipated in online PD or study groups — 
are used in this report. The study guided reeommendations for praetitioners implementing 
blended PD models tied to a eoneeptual map of faetors affeeting sueeessful delivery of the PD. 
Lessons learned from the study inelude lessons relating speeifieally to online PD and others 
related to PD in general. 
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Lessons Learned Implementing Online Teacher Professional 
Development within a School Improvement Initiative 

Introduction 

Raising teacher effectiveness is central to school improvement initiatives and educational 
reforms. Current policy initiatives, including Race to the Top, are focused on teacher 
effectiveness as one of the levers promoting effective school improvement and the larger system 
goal of ensuring college and career readiness for all students. School improvement often entails 
significant professional development (PD) for teachers tailored to standards, curriculum, and 
assessments. Ongoing PD is important for exposing teachers to new teaching strategies and 
providing opportunities to share effective teaching practices. Teachers agree that PD is vital to 
school improvement; a large-scale, nationally representative survey of over 40,000 teachers 
conducted found that 85% of the teachers viewed PD as “absolutely essential” or “very 
important” to retaining good teachers (by comparison, 81% of teachers viewed higher salaries as 
“absolutely essential” or “very important”) (The Bill and Melinda Gates Foundation, 2010a). 

The traditional model of PD is based on face-to-face delivery. The number and 
complexity of learning objectives needing to be addressed with PD, combined with the number 
of teachers in training, lead to substantial costs for PD. These costs present a significant barrier 
for school systems, impeding their ability to raise teaching quality. This barrier is particularly 
formidable in today’s climate of school budget cuts, which have led to fewer funds for PD, 
travel, and substitute teacher pay. 

In order to take quality teaching programs to scale, we need to reduce the cost and 
increase the efficacy of PD. Online delivery offers a means of reducing costs substantially; 
recent work points to the potential of online delivery of PD to yield cost efficiencies and produce 
results that are comparable if not better than face-to-face PD methods (U. S. Department of 
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Education, 2010). Moreover, it is possible that large-seale online PD may introduee effieieneies 
that make the training more focused, achieve a more job-embedded approach, and increase 
ongoing collaboration among participants and facilitators, thereby strengthening the support for 
professional learning eommunities and the benefits that aeerue from them (Lord, 1994). Reeent 
studies have touted the benefits of blended PD solutions that eombined traditional faee-to-faee 
methods with online delivery. As Hidalgo (2010) states in Face-to-Face V5. Online Professional 
Development? Do Both! The Power of the Blended Model, “Blended models ean eombine faee- 
to-faee sessions with several online follow-ups that give teaehers opportunities to get expert and 
peer adviee on eurrent instruetional issues, when they need it, in small inerements, and eonneeted 
to what they are teaehing.” In its meta-analysis of evidenee-based praetiees in online learning, 
the U.S. Department of Edueation (2010) also found that instruetion eombining online and faee- 
to-faee elements was advantageous to purely faee-to-face instruetion or purely online instruetion. 

ACT and Ameriea’s Choiee, with funding from the Bill & Melinda Gates Eoundation, 
joined forees to deliver blended models of PD and to identify praetieal suggestions for removing 
barriers to suceessful delivery. A qualitative evaluation of the projeet used teaeher surveys and 
interviews to measure pereeptions of both online and faee-to-faee offerings within blended 
models of PD. The PD was delivered during the 2009-2010 sehool year, with evaluation from 
fall 2009 to fall 2010. 

An Integrated School Improvement Initiative 

Rigor & Readiness was a school improvement partnership among ACT, Ameriea’s 
Choiee, and sehool distriets designed to inerease the number of students graduating high sehool 
who are eollege and career ready. ACT and America’s Choice supported partner distriets’ 
implementation of a rigorous and eoherent eollege and career readiness system that is eonsistent 



3 


with the educational systems of the highest performing countries. The initiative’s design stressed 
comprehensiveness and close alignment of system components. Its cornerstone was ACT’s 
research-based College Readiness System, which defines the performance levels and standards 
needed for college and career readiness, as measured by a sequence of assessments from grades 7 
through 12. Aligned with these assessments was a rigorous core curriculum with end-of-course 
examinations aligned to college and career readiness standards, safety net programs for students 
who need help accessing the core curriculum, and support for students’ psychosocial 
development. 

In the design of Rigor & Readiness, PD was seen as a crucial lever for raising college and 
career readiness. The initiative stressed the critical role of teachers, administrators, and coaches 
in raising student achievement — and the belief that teachers’ skills are enhanced by continuing 
PD throughout their careers. PD supporting the core curriculum, safety net programs, and 
students’ psychosocial development was central to the initiative. 

Description of Programs Supported with Online PD 

The online PD supported implementation of key components of Rigor & Readiness, 
including the core mathematics curriculum for 9th grade (QualityCore Algebra 1), a safety net 
English language arts program for 6th- and 9th-grade students who were two or more years 
behind grade level in reading comprehension (Ramp-Up Literacy), a program for students in 
grades 6-9 who need help accessing the core mathematics curriculum (Mathematics Navigator), 
and a system for monitoring students’ psychosocial development in support of 9th grade teacher 
teams and other small learning communities. To enable better understanding of the context of the 
study, we first describe these programs in greater detail. 



4 


QualityCore Algebra 1 is an instructional improvement program designed to raise the 
rigor and quality of Algebra 1 courses. Instead of a specialized curriculum, QualityCore offers 
supports for curriculum and instruction, including instructional resources (course standards, test 
blueprints, and model instructional units), a formative item pool and test builder linked to course 
standards, and an end-of-course assessment. Model use of QualityCore would include a 
collaborative “gap” analysis by teachers and instructional coaches to compare existing course 
standards to QualityCore standards (course objectives) to identify areas that need more attention 
or could otherwise be strengthened by utilizing QualityCore’s resources. A key feature of 
QualityCore is that formative items are offered at varying depth of knowledge levels, allowing 
educators to assess — and then target instruction towards — students’ higher-order thinking skills 
that are critical for future success. 

The Ramp-Up Literacy program is a full English language arts (ELA) replacement 
curriculum, integrating reading comprehension and writing instruction across narrative, 
explanatory, and persuasive genres. The curriculum typically is delivered in double-period 
courses that seek to accelerate the learning of students who are two or more years behind grade 
level in English language arts. Ramp-Up Eiteracy courses follow a workshop model of 
instruction in which teachers provide a cycle of behavioral monitoring and guided practice using 
instructional routines and classroom rituals. Routines are the components of the course that 
determine what happens, in what order, and for how long for each day of instruction. An 
example routine is the work period — during which time students work apart from the teacher, 
either in pairs, small groups, or independently. Routines help the teacher to focus instruction 
through whole-group lessons, small-group reading instruction, small-group strategy lessons, or 
individual and small-group reading and writing conferences. Classroom rituals are prescribed 
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ways of supporting Ramp-Up’s routines and provide consisteney and reliability for the students 
and teaeher; they help teaehers manage transitions among routines with efficieney and they help 
students build independence as learners. Ramp-Up Literacy is designed to offer students an open 
and non-threatening learning atmosphere. The routines and rituals promote organization and 
good habits, productive use of time, and student responsibility for learning. 

Mathematics Navigator works by correcting student misconceptions, using formative 
assessment strategies designed to continually monitor student work and identify misconceptions. 
It is offered to students who are enrolled in on-grade level math courses, but who struggle with 
specific math concepts. Mathematics Navigator targets specific content gaps that students need 
to master to be successful moving forward. It is intended to augment the school math curriculum, 
and can be delivered during school hours, tutoring, after-school, weekend, or summer programs. 
Mathematics Navigator is offered in a series of discrete modules. Students enter Mathematics 
Navigator on an as-needed basis and exit after demonstrating mastery, as measured by growth on 
embedded pre- and post-tests, of the targeted concepts. 

In addition to its academic supports. Rigor & Readiness also provides a system of 
psychosocial supports, featuring behavioral monitoring systems and interventions designed to 
help educators respond to students’ identified psychosocial needs. As part of this study, school- 
based teacher teams for 9th grade students were trained to use a system of assessments, including 
the ENGAGE Grades 6-9™ (formerly known as the Student Readiness Inventory), a student 
self-report inventory; and the ENGAGE Teacher Edition (formerly known as the Behavioral 
Monitoring Scales), an instrument teachers use to rate student behavior. Both assessments 
measure academic behaviors in three broad domains: motivation, self-regulation, and social 
engagement. The training was developed with the belief that correctly interpreting and using 
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assessment results is a key step towards responding to students’ needs — and developing or 
directing interventions to meet those needs. 

Description of Online PD Models Implemented 

The online PD was delivered using a variety of methods that can be categorized as online 
seminar, online tutorial, online collaboration, or online libraries. 

Online seminars were designed for small- to medium-size groups and maintained the key 
elements of face-to-face workshop-style seminars. They featured a blend of pre-work (reading 
and viewing selected videos prior to the seminar), direct instruction using collaborative 
whiteboards for webinars, modeling through the delivery of streaming video, checkpoints for 
knowledge, and small-group collaboration around student work (using the collaborative 
whiteboards). In general, online seminars were used to replace specific onsite training days that 
would normally have been scheduled during the academic year as follow-ups to an initial 
summer seminar. In this way, online seminars were linked to a larger blended “course” model 
that occurred over the year. 

Online tutorials offered targeted coaching and one-to-one interactions using the 
collaborative whiteboards*. The interactions included real time discussion between an 
experienced tutor and the teacher, and tools that allowed the teacher and tutor to share their 
writing and electronic files. This model included an introductory session in which the online 
tutor reviewed elements of the program and answered questions. After a short interval, during 
which the teacher began implementing the program, there was another session in which the tutor 


* A whiteboard is a web-based collaboration environment that allows multiple users to view and edit the same screen 
at the same time. In this study, the primary collaboration environment used was Adobe ConnectPro, although the 
principal features used are available in almost all such competitive products. 
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answered questions and guided the teaeher in a reflection on progress to date. A third session 
involved analyzing student work generated from implementation of the program. 

Online collaboration tools and protocols that support professional learning communities 
as a professional development opportunity were also introduced. These collaboration tools 
included social networking features (blogs, shared commentary), threaded discussion boards, and 
online journals where participants could reflect on what they learned in training — and on what 
they implemented with their students. 

Teachers also had access to online libraries including videos of instruction, student work 
with commentary, a research library with relevant whitepapers, and other tools for implementing 
the supported programs. The online libraries were indexed and sequenced to imply a learning 
path that builds knowledge through the user’s independent, self-paced use of the materials. 
Included in the online libraries were materials designed to facilitate face-to-face teacher team 
interactions; we refer to these resources as study groups. 

Applications of Online PD Models Implemented 

Online Seminars 

The online seminar approach was applied to the PD supporting the two full-year 
programs — QualityCore Algebra 1 and Ramp-Up Literacy. The baseline PD for each consisted 
of an initial summer institute continued through follow-up training days throughout the academic 
year. This continual connection provided a good opportunity to blend online with face-to-face 
delivery as described below. 

QualityCore Algebra 1 PD typically consisted of three days of face-to-face training 
during the summer followed by two discrete days of face-to-face training during the academic 
year, usually one in the fall and one in the winter or spring. In this study, we used online delivery 
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for the specific content of the final face-to-face training day. This was easy to do within the PD 
design as the original face-to-face follow-up days are structured around short segments of 
content (modules). In this study, the content from the modules used for the final face-to-face 
training day was streamed out during the spring semester in a mix of webinars for small groups 
as well as individual coaching sessions. Online PD offerings for the final day of QualityCore 
Algebra 1 training began in January 2010 and continued through early May 2010. The topics 
covered included; scaffolding assessments to match instruction; looking at student work (using 
QualityCore’s resources for systematically examining student work); and revising lessons for 
rigor and relevance. Online seminar was the primary method of delivery, with online tutorials 
used as a follow-up to the seminars. 

Ramp-Up Literacy PD provided teachers implementing the course with eight days of 
training. These eight days typically are delivered as four initial days of face-to-face training in 
the summer followed by four days of training during the school year — two days in the early fall 
and two days in the winter or spring. As opposed to simply replacing one or two days of face-to- 
face follow-up training with online delivery of the same, the online PD offerings for Ramp-Up 
Literacy were designed to deliver the content of the follow-up training that explicitly supported 
the work period routine. This was chosen because the work period is often one of the most 
challenging aspects of implementation for teachers new to Ramp-Up. The online delivery began 
in December 2009 and continued through early May 2010. A series of three connected online 
seminars were featured, and the Ramp-Up Literacy design was modified to include journals, 
threaded discussion, and virtual “office hours” for individuals or small groups. The topics 
covered in the online seminars included; implementing the work period, using formative 
assessments to group students for independent and small-group activities, methods for looking at 



9 


student work, and building class profdes for small-group instruction delivered by the teacher. 
This design allowed for the facilitators to gain visibility into the classrooms using Ramp-Up so 
that they could monitor implementation and provide coaching support to teachers across the 
winter and spring. 

Online Tutorials 

Mathematics Navigator training is typically delivered as a brief “how to” orientation 
focused on the instructional design and materials. Sometimes, instructors are asked to implement 
the program with no orientation. This approach derives from its use as a short-cycle intervention 
that augments the regular curriculum and that is often taught by non-specialists, 
paraprofessionals, and/or volunteers in addition to mathematics specialists. Orientations are 
typically delivered in advance of the school year as a half- or full-day face-to-face training 
session. Often, training occurs at a time that is convenient for stand-alone PD but not necessarily 
optimal for supporting the start of the program (e.g., instructors are trained weeks prior to 
receiving materials or having any students assigned). Thus, a more flexible online support was 
seen as a promising approach. As such, the online PD for Mathematics Navigator was delivered 
through a series of online tutorials that were designed to occur (a) just prior to use, (b) after the 
first five lessons in order to address any implementation barriers, and (c) following the first 
formative assessment checkpoint, to help teachers use assessment data effectively throughout the 
remaining 12-15 lessons. To participate in this training, teachers used the collaborative 
whiteboard and a headset to communicate with tutors, who were skilled mathematics teachers 
trained by America’s Choice in the delivery of Mathematics Navigator. This method allows 
tutors to see and respond to teachers’ work and to carry on live discussions with the participating 
teachers. Tutors used the technology to (a) introduce teachers to the overall structure of 
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Mathematics Navigator (replacing the traditional face-to-face introduction), (b) help teachers 
with the PD-targeted strategies as they work through lessons, and (c) analyze student work and 
assessment results to unearth and correct misconceptions. 

Online Collaboration Tools and Online Libraries 

Support for teacher teams was the primary application of online collaboration tools. The 
context, training, and analysis of data relating to psychosocial assessment was supported with 
online PD through online collaboration tools and online libraries. 

Teacher teams or professional learning communities (PLCs) have been shown to be a key 
mechanism for improving teacher effectiveness and instructional leadership (Lord, 1994). Within 
the scope of teacher teams is the “small learning community”; in particular, where a small group 
of core academic teachers share the teaching responsibilities for a specific group of students. In 
this way, teacher teams create a personalized learning environment for their students and can 
work together to provide support across disciplines and within the area of academic behavior 
(psychosocial supports). 

In February 2010, teachers organized into teams were introduced to a virtual 
collaboration environment and guided through the use of tools and expectations for 9th grade 
teacher teams. The teacher teams had representation from the English language arts, 
mathematics, science, and social studies faculty members who work with a defined group of 
students, usually ranging in number from 20 to 80. A screenshot of the online team environment 
is presented in Figure 1. Study groups, which are downloadable materials that organize and 
facilitate learning objectives for face-to-face teacher team meetings, were developed for specific 
topics, including interpreting and using data from the student behavioral assessment (ENGAGE) 
and student behavioral ratings made by teachers (ENGAGE Teacher Edition), using academic 
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data (from ACT’s EXPLORE assessment) to make decisions about students, and using resources 
for students’ career exploration and planning. The goal of the study groups and online tools was 
to promote teacher team collaboration around psychosocial and academic data to understand the 
needs of students, and to help teachers work better as a team. ENGAGE was administered to 
approximately 5,500 9th grade students in fall 2009 or January 2010; the ENGAGE Teacher 
Edition assessments were collected from teachers for a subset of over 600 of these 9th grade 
students in Eebruary 2010. Data from these assessments were used in the teacher team meetings; 
the study group materials acted as a curriculum for the teacher team meetings focused on the 
assessment results. While use of the study group materials does not require online PD, it was 
included in our evaluation as a potentially effective and scalable training model for improving 
teacher teams. 

FIGURE 1. School-Based Teams Study Group Environment 
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Research Methods 

Development of Conceptual Map 

The original study plans were built around Kirkpatriek’s (1996) four-level model for 
measuring training effeetiveness. The four levels are Reactions (partieipant satisfaetion and 
pereeived relevanee of the training program to their everyday work), Learning (the extent that 
training partieipants have advanced in skills, knowledge, or attitude). Transfer (whether the 
newly acquired skills, knowledge, or attitudes are being used in the everyday environment of the 
training participant), and Results (the success of the training program in terms of whatever 
outcome it was conceived to achieve). According to this model, findings from each prior level 
serve as a base for the next level's evaluation. Thus, each successive level represents a more 
precise measure of the effectiveness of the training program. 

An initial wave of data collection, in fall 2009, included classroom observations and 
teacher surveys ascertaining program implementation and satisfaction with face-to-face training. 
This wave of data collection, along with initial experiences designing and implementing the PD, 
gave the research team a better understanding of the schools, programs, and teachers that were 
the subject of the PD. After these initial experiences, the study plans were revisited to target the 
research towards questions that were most important and most likely to be informed by the study. 
The initial conceptual map, presented in Figure 2, was created to better organize the research 
inquiry and to help prioritize and guide data collection efforts. The map is intended to organize 
supports and barriers of effective PD, as well as outcomes of PD delivery. The arrows between 
the map’s components represent expected directionality of effects. For example, the map posits 
that contextual factors influence training logistics and teachers’ expectations about the training; 
and contextual factors, training logistics, and teachers’ expectations all influence decisions to 
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participate in training. The map’s components, as conceptualized by the research team, are 
described in Table 1. 

FIGURE 2. Initial Conceptual Map 
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TABLE 1 

Conceptual Map Component Descriptions 


Component Name Component Description 


Supports and Barriers - Contextual Factors 

General school improvement 
climate 

The extent to which a shared vision of school improvement, supported by 
district leadership, exists between school leaders and faculty and staff. School 
leaders and teachers work towards shared goals that are measurable and 
attainable. 

School/classroom climate 

The qualities of the school and classroom environment that promote learning, 
as well as teachers’ ability and willingness to improve teaching. This includes 
the extent to which students feel safe and the extent to which the school and 
classroom are free of disruptions. 

Leadership support 

The support provided to teachers by school leaders in pursuit of a common 
goal. Leadership support entails giving feedback, support, and encouragement 
through effective communication, ample time for learning and collaboration, 
and support of teachers’ experimentation with new strategies. 

Colleague support and 
involvement 

The degree that educator colleagues provide mutual support for participating in 
training, reflecting on teaching and how the training informs their practice, and 
transferring learning to practice. Colleague support and involvement also 
represents the degree that educators support each other’s teaching endeavors, 
in general, and implementation of new teaching strategies, in particular. 

Individual/teacher commitment 

The extent to which teachers are committed to school improvement efforts 
through learning and implementing new programs. 

Availability of training and 
classroom resources 

The extent to which training materials and classroom resources associated with 
the training are made available to teachers in a timely manner for use during 
both training and implementation. 

On-going support for training 
and implementation 

Teachers’ receipt of continued support from school leadership and training 
providers throughout the year for both training and implementation. This 
includes timely access to experts, resources for successful implementation, and 
adequate time for incorporating new ideas. 

Supports and Barriers - Logistics 

Logistical convenience 

The extent to which scheduled training times offer convenience and flexible 
times for teachers to attend training. 

Time availability 

The extent to which teachers are provided ample time to attend training 
without adversely affecting their teaching. 

Knowledge of training topics and 
expectations 

The extent to which teachers have been adequately informed of upcoming 
training activities. This also includes understanding of the purpose and nature 
of the training activities and expectations for attendance and engagement. 

Supports and Barriers - Expectations/Experience 

Individual differences 
(motivation, teaching 
experience) 

Personal characteristics of teachers that affect their disposition towards PD. 
These include attitudes, personality factors, teaching experience, and personal 
circumstances that affect their ability to attend training. For example, one 
teacher may be motivated by professional responsibility, while another teacher 
may feel that all PD is a waste of time. 

Comfort and experience with 
technology 

Teachers’ level of comfort and experience with the technology needed for 
online training derived from their prior training or experience with the training 
tools. 

Expected value of training 

The level to which teachers believe that the planned training activities are 
worthwhile and will improve their teaching ability. 

Satisfaction and Engagement 

Attendance and participation 

The degree to which teachers not only show up for training, but also actively 
participate in training activities. 
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Delivery format differences 

The extent to which teachers find value and satisfaction with different delivery 
formats, such as online seminars, one-to-one tutoring, collaboration, or 
whiteboard activities. 

Engagement and retention 

The level to which teachers feel engaged in and are able to learn from the 
training activities, increasing the chances they want to return for more training 
sessions. 

Perceived satisfaction and value 
of training 

The extent to which teachers feel satisfied with the quality of training they 
received, as well as the extent that they believe the training was worthwhile 
and will improve their teaching. 

Learning, Transfer, and Ontcomes 

Change in attitudes, beliefs, 
knowledge 

The extent to which teachers’ attitudes, beliefs, and pedagogical content 
knowledge change as a result of the training they receive. 

Transfer to practice 
(implementation) 

The degree to which teachers are able to transfer what they learn in training to 
their teaching activities and responsibilities in and out of the classroom. 

Improved student learning 

The extent to which students’ learning gains improve based on PD-targeted 
strategies teachers have learned and implemented in their classrooms. 


While the initial eonceptual map retained components of Kirkpatrick’s (1996) 
framework, it was acknowledged that the research was not designed to attribute differences in 
certain components (teacher learning and transfer; student learning outcomes) to differences in 
training experiences or the support and barrier components. It is more feasible to measure 
proximal components, such as teachers’ reactions to the PD, than more distant components such 
as changes in teacher and student behavior. Other work that informed the map development or 
support its structure include: Guskey’s (2002, 2000, 1999) work guiding evaluation of PD, and 
work by Knapp (2003) discussing the importance of teacher support, in terms of resources, time, 
and policies built on the factors related to effective PD. Knapp also discussed the role of the 
school context, nested within a larger policy context, which plays a role in teacher learning. 
Ingvarson, Meiers, and Beavis (2005) developed a theoretical framework for investigating 
factors related to the impact of PD on teacher and student outcomes, suggesting that the impact 
of PD was related to background variables pertaining to the teachers and school environment, 
structural features of the PD such as amount of time and amount of participation, and opportunity 
to learn, mediated by the professional community within schools. Several studies cite the 
convenience of online PD in terms of flexibility of time, opportunities for self-paced instruction. 
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access to experts and other teachers outside of the sehool or district, and opportunities to 
communicate after the conclusion of the PD (Carey, Louis, Kleiman, Russell, & Venable, 2008; 
Dominguez, Nieholls, & Storandt, 2006; Liu, Carr, & Strobel, 2009; Russell, Carey, Kleiman, & 
Venable, 2009). It has also been suggested that partieipants may be less inhibited when 
partieipating in online PD than they would be in a face-to-faee situation (Carey, et al. 2008), and 
that school systems realize the cost effectiveness of online over face-to-face PD (Holmes, Signer, 
& MacLeod, 2010). 

The initial eoneeptual map was developed in winter and spring of 2010 and used to guide 
the creation of end-of-year teaeher surveys (given in May and June, 2010) and follow-up 
interview protoeols completed in November 2010. 

Research Questions 

To further help guide the data colleetion efforts, broad researeh questions of interest were 
articulated after creating the initial eoneeptual map. As deseribed next, survey and interview 
questions were designed to inform these questions. 

• What are the environmental supports that need to be in place in order for online PD to 
be most effective? 

• What factors affect teacher partieipation in, and level of engagement in, online PD? 

• Relative to traditional PD, were teachers satisfied with the quality and relevance of 
the online PD? 

• Did the online training enhance teachers’ perceived knowledge of training-targeted 
coneepts? 

• Did teaehers report being able to apply what they learned in online training to their 


work? 
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• Do teachers expect the online training to result in improved student learning? 

Data Collection 

Teacher Surveys. Development of surveys administered at the end of the 2009-2010 
academic year was informed by the conceptual map and research questions. Two forms of 
surveys were developed — one for teachers who participated in online training {online 
participants) and one for teachers for whom online PD was offered, but who did not participate 
{online nonparticipants). With the exception of one item that addressed reasons for 
nonparticipation, the items on the survey form for online nonparticipants were a subset of the 
items on the form for online participants (the additional items addressed experiences with online 
PD). 

The teacher surveys were paper & pencil forms. At the end of the 2009-2010 school 
year, the surveys were distributed to teachers by America’s Choice and ACT personnel who 
worked within the districts as part of the Rigor & Readiness initiative. The surveys were 
distributed along with postage-paid, addressed envelopes and were returned to ACT for data 
entry. At the time of survey distribution, teachers were asked whether or not they participated in 
online PD — and their response determined which survey form they received. Teachers were 
assured that their individual responses would be kept confidential and that data would only be 
reported in aggregate form. Teachers were not provided special incentives for completing 
surveys. 

Most of the survey items asked teachers to indicate their level of agreement with 
statements using a 6-point scale, where 1 = strongly disagree, 2 = moderately disagree, 3 = 
slightly disagree, 4 = slightly agree, 5 = moderately agree, and 6 = strongly agree. For example, 
online participants and nonparticipants were asked to rate their agreement with: “1 understood 
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why I was asked to participate in online PD.” Another 6-point scale ascertained satisfaction with 
different aspects of PD. For example, “Access to experts outside of PD” was rated as 1 = very 
dissatisfied, 2 = moderately dissatisfied, 3 = slightly dissatisfied, 4 = slightly satisfied, 5 = 
moderately satisfied, and 6 = very satisfied. This was rated separately with respect to face-to-face 
and online PD. Other items were rated on a 4-point scale, where 1 = did not influence, 2 = 
slightly influenced, 3 = moderately influenced, and 4 = strongly influenced. For example, 
teachers were asked to rate the extent that certain factors influenced decisions to participate in 
PD, such as “location convenience.” Again, this item was rated separately with respect to face- 
to-face and online PD. The survey data are summarized and presented in Appendix A. 

Structured Interviews. Structured interviews were developed and conducted with teachers 
and non-teachers (training facilitators, training coordinators, and program leaders) in the fall of 
2010. The interviews assessed many of the same topics addressed by the survey, but at a deeper 
level — and with the potential to probe interviewees for more information. Some questions were 
open-ended, which allowed interviewees to reveal information we may have not known to ask 
about. By conducting the interview in the fall of 2010, teachers were able to provide information 
about the ongoing effect of the previous year’s PD. 

Interviews were first conducted with the non-teachers, which helped shape the questions 
on the teacher interviews. Two forms of teacher interviews were used. One form applied to 
teachers implementing the academic programs (QualityCore Algebra 1, Ramp-Up Literacy, and 
Mathematics Navigator). Another form applied to teachers who used the study groups within 
their teacher team meetings. Non-teachers were invited to participate in the interviews through 
emails and phone calls. Teachers were invited to participate with letters through regular mail. 
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and with follow-up email invitations. Teachers were given $100 for participating in the interview 
and were assured that their interview responses would be kept confidential. 

Early in the teacher interviews for the academic programs, the interviewer determined if 
the teacher was an online participant or a nonparticipant. The flow of the interview and many of 
the specific questions asked were different for online participants and nonparticipants. 
Nonparticipants were asked about barriers to their participation in online training; for example, 
“Were there any scheduling conflicts that prevented you from attending online PD?” and “What 
would have made you more likely to participate in the online training?” Unlike nonparticipants, 
online participants were asked questions about their experiences with the online training, such as 
“What suggestions do you have for improving the online training?” and “Are there problems or 
barriers that have kept you from applying what you learned in online training to your teaching 
duties?” In lieu of these questions, nonparticipants were asked similar types of questions about 
their experiences with face-to-face training. 

The interviews were conducted by four members of the project evaluation team and 
transcribed to electronic documents. To identify responses with common themes, a coding 
system was developed and applied to the transcribed data using software for the analysis of 
qualitative data (Muhr & Friese, 2004). An initial coding system was developed after review of 
the non-teacher interviews, and was informed by the conceptual map and our cumulative 
experience with the project prior to the teacher interviews. The coding system was revised after 
completion of nearly half of the teacher interviews (but before being applied to the transcribed 
interviews), and underwent smaller revisions through the course of coding the transcribed 
interviews. The interview data are summarized and presented in Appendix A, and the coding 
system is presented in Appendix B. 
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Reporting Conventions. Data are presented throughout the report for online partieipants, 
nonpartieipants, and in some eases both groups combined. Some survey and interview data 
pertain to experiences with online PD, some pertain to face-to-face PD, while others ask PD- 
related questions but do not distinguish between the two modes. Other questions do not pertain 
directly to PD but rather assess school climate, relationships with other teachers, and comfort 
with technology. Interview data for the teacher team study group participants are reported 
separately from the data from the three academic programs because the training models differed 
considerably (study groups facilitated face-to-face interactions of teacher team members, while 
the online PD for the academic programs used online seminar, online tutorial, and online 
collaboration tools and resource libraries) and because a different interview form was used for 
the study group participants. 

The survey data referenced in this report are calculated means and percentages of 
teachers agreeing with certain statements. For example, the mean response (using the 1-6 
agreement scale) to “I often have difficulty choosing what to do in my classroom after hearing 
about so many best teaching practices” was 3.1, indicating that surveyed teachers “slightly 
disagreed” with the statement, on average. Meanwhile, 42% of teachers agreed to some extent 
with the statement. 

The interview data are presented in this report as percentages (either absolute or using a 
qualifying descriptor of the percentage) of teachers who expressed a certain theme. We also use 
teacher (and non-teacher) quotes that exemplify certain themes. Qualifying frequency descriptors 
are often used in place of reporting actual percentages using the conventions described in Table 


2 . 
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TABLE 2 

Qualifying Descriptors of Frequencies Used to Present Interview Data 


Descriptors of teacher 
frequency 

Descriptors of 
theme prevalence 

Percentage 

range 

Example 

Nearly all 

Nearly unanimous 

75-100% 

Nearly all of the teachers interviewed indicated a 
medium or high level of comfort with technology 
and participating in online training. 

Most 

Prominent 

50-74% 

Most interviewed teachers found the online PD to 
be engaging. 

Some 

Common 

25-49% 

Another commonly cited positive aspect of online 
PD was being able to communicate and share with 
others outside of the school or district. 

Few, 

a small number of 

Less common 

10-24% 

Though less common, setting clear expectations 
was suggested by interviewed online participants 
as a way to improve online PD. 


Program, Training, Survey, and Interview Participants 

Summary counts of the number of teaehers partieipating in eaeh program, ineluding those 
who partieipated in online training and those who did not, are presented in Table 3. Also given 
are the numbers of teaehers providing survey data, interview data, or either form of data (N). The 
eounts in the Total eolumns are estimates based upon program training rosters, reports from 
training faeilitators and program leaders, and reeords of partieipants using online training 
resources. The counts in the N, Survey, and Interview columns are based on the data eollected 
from teaehers in the surveys and interviews. From Table 3, we see that the pereentage of teaehers 
that partieipated in online training varied significantly across the three programs from 18% for 
QualityCore Algebra 1, 53% for Math Navigator, and 68% for Ramp-Up Literaey. Beeause we 
only reeeived data for teaehers that partieipated in the study groups supporting teaeher teams — 
and did not have an indieation for whieh teaehers were offered study groups but ehose not to 
partieipate — the eolumns for Online Nonparticipants are not applieable for the teaeher team / 
study group program. The small pereentage of online partieipants for QualityCore Algebra 1 is 
due primarily to a low participation rate (9 of 96 teaehers, 9%) in the district with the most 
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teachers implementing QualityCore Algebra 1. In this district, teachers were going to have the 
final day of face-to-face training regardless of whether or not they attended the online training, 
and thus the online training was not deemed important. 

TABLE 3 

Survey and Interview Participants 


Program 

Online Participants 

Online Nonparticipants 

Total 

Nnmber of 
Program 
Participants 

Total 


Snrvey 

Interview 

Total 

Z 

Snrvey 

Interview 

QualityCore 
Algebra 1 

27 

13 

9 

8 

125 

22 

18 

4 

152 

Math Navigator 

62 

14 

5 

11 

54 

8 

6 

2 

116 

Ramp-Up 

Literacy 

36 

17 

7 

11 

17 

17 

7 

12 

53 

Teacher team / 
Study Groups"' 

34 

13 

8 

6 

NA 

NA 

NA 

NA 

34 

Total 

148" 

51"’" 

29 

36 

191" 

47^ 

31 

18 

329" 


'Number of program participants with survey or interview data. 

^Total is less than the sum across rows because some teachers were participants in multiple programs. 

^The total number of teachers with interview or survey data is 92; six teachers were online participants for one 
program but nonparticipants for another program. 

"'Use of study group materials is not considered online PD, but is reported here along with online participants to 
capture the number of teachers who participated in a nontraditional form of PD. 

Overall, 92 individuals contributed survey, interview, or both sources of data towards the 

three academic programs or teacher team study groups. Nine non-teacher interviews were 

conducted with personnel external to the school systems — including four content experts who 

delivered the online PD (training facilitators), two training coordinators, and three program 

leaders who designed the online PD systems. Overall, data from 101 individuals are referenced 

in this report. 

Among the 60 teachers who completed the end-of-year survey, 29 were online 
participants and 31 were nonparticipants. Teachers in both groups tended to be experienced 
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teachers, with 76% and 81% of online participants and nonparticipants, respectively, having at 
least four years of experience. Only 10% of surveyed teachers had less than two years of 
teaching experience. 

Fifty-four teachers participated in the teacher interviews, including 36 online participants 
and 18 nonparticipants. Among the 54 teachers interviewed, 12 implemented QualityCore 
Algebra 1, 23 Ramp-Up Literacy, 13 Mathematics Navigator, and six participated in study 
groups. Overall, the sample of teachers interviewed were experienced teachers, although there 
were more teachers with five years of experience or less who participated in online PD (41%) 
relative to the nonparticipants (26%). Nearly half of the teachers in both groups had more than 
10 years of teaching experience, with 49% and 58%, respectively, of online participants and 
nonparticipants reporting more than 10 years experience. Nearly all teachers interviewed 
indicated that they had also received face-to-face PD for their respective program. 

Among surveyed teachers, 80% reported attending at least three sessions of face-to-face 
PD; just one reported attending no face-to-face PD sessions. Among the surveyed online 
participants, 58% reported attending at least five hours of online PD and 16% reported attending 
less than two. Most interviewed online participants attended at least three online PD sessions, 
while about one-third attended just one or two sessions. All interviewed nonparticipants reported 
attending at least three sessions of face-to-face PD. 

Study Limitations. Our data are based on teacher self-report, which can be inaccurate due 
to respondents’ inability to recall prior events, or their inclination to provide socially-desirable 
responses. The PD was delivered during the 2009-2010 school year, the surveys were 
administered at the end of the school year, and the interviews were conducted during the fall of 
2010; therefore, memory decay is probably not a large source of error. Social desirability bias is 
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potentially more of a concern; teachers have been shown to over-report positive changes in 
classroom practices, which is attributed to a desire to appear favorably in relation to one’s peers 
(Kopcha & Sullivan, 2006). However, the extent to which participants feel that their responses 
are private can help reduce this bias (Scaeffer, 2000). Teachers were told both in the survey 
instructions and at the start of the interview that their responses would remain anonymous; 
therefore, while the possibility of social desirability bias should not be overlooked when 
interpreting the results of this study, steps were taken to try to mitigate this problem. 

As shown in Table 3, we were only able to collect survey or interview data on about one- 
third of the online participants and about one-fourth of the nonparticipants. It is possible that the 
sample of teachers who did not respond has different perceptions of the programs and training. 
With a relatively small sample of teachers within each program and overlap of online delivery 
methods across programs, we have little power to attribute differences in teachers’ perceptions to 
program-specific training content. Additionally, teacher background and demographic 
information was provided for only about 50% of the participants. Another limitation is that the 
study was conducted in four large school districts that have historically low aggregate 
achievement and that are undergoing extensive school improvement reform; it is therefore 
possible that some of our findings are specific to this context. The study evaluated PD programs 
delivered by one organization and it is possible that there are nuances to our findings that may 
not be encountered by other PD providers. As described earlier under the development of the 
conceptual map, the study was not designed to attribute differences in teachers’ training 
experiences to objective measures of teacher knowledge, implementation, or student learning 
outcomes. Aside from teacher perceptions, the effectiveness of the training facilitators, and the 
quality of the PD itself were not assessed using other objective measures. Another limitation is 
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that there was no formal eost analysis to determine whether the potential cost savings offered by 
online PD was realized. 

Results and Lessons Learned 

Our study unearthed some practical suggestions for improving the delivery of blended PD 
models. While some of the findings are specific to online PD, others are applicable to any PD 
endeavor. To present results, we grouped common findings together into seven lessons learned. 
Based upon these lessons learned, a revised conceptual map will be presented, along with 
specific recommendations for the design and implementation of blended PD models. 

Lesson 1: Synchronous forms of online PD suffered from coordinating difficulties 

Live, or synchronous, forms of online training can be hindered by scheduling and 
coordinating difficulties. Asynchronous forms of training may reduce the need for common 
training time and alleviate scheduling difficulties; however, they also eliminate the community, 
collaboration, and individualization offered by synchronous forms. Results from the study 
indicate that online training can be more difficult to schedule and coordinate than face-to-face 
PD, particularly when regular teacher PD days are not utilized and teachers must use their 
planning periods or class time to participate. In order for synchronous online PD to run 
smoothly, multiple teachers need to be scheduled during the same time, all computers and 
peripherals must be running properly, and if regional PD is desired, then schedules would need 
to be coordinated across multiple districts. 

Among surveyed teachers, over half indicated that they typically participated in face-to- 
face PD during teacher work days (51%), whereas when they participated in online PD was more 
variable, including during regular planning time (26%), teacher work days (42%), or other days 
(32%). Similarly, interviewed teachers who participated in online PD indicated that they did so 
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mainly during school hours other than the planning period (39%), or during the planning period 
(35%). Two of the interviewed teachers mentioned a laek of common planning periods within 
their sehool as an impediment to participating in the online PD, and one of the training 
coordinators interviewed said that differenees in schedules aeross sehools (e.g., 60- vs. 90- 
minute elass periods) made seheduling very difficult. Surveyed teaehers slightly disagreed, on 
average, that online PD was easier to schedule than face-to-face PD (mean=3.0 on the 6-point 
seale). 

Aeeording to the non-teaehers interviewed, distriets realized during the sehool year that 
state funding was in serious trouble and they were going to have a diffieult time paying for 
substitute teaehers. Adding to the faet that substitutes were not going to be provided, most 
districts had determined that online training was only to be offered during normal sehool hours. 
In some cases, teaehers used their planning periods to attend training or to eover other teaehers’ 
classes so they eould attend training. It was eommon for teachers to miss either the beginning or 
ending of training sessions due to elassroom coverage issues. In a few instances, teachers had to 
split their students aeross more than one classroom to keep from overburdening the teachers 
covering for them. 

Teaehers interviewed also indieated that classroom coverage created seheduling and 
partieipation problems; most of the teaehers surveyed indieated that substitute teaehers were 
needed to teaeh while they participated in online PD. However, substitute teaehers were often not 
available. A lack of time or substitute teachers was mentioned as a disadvantage of online PD by 
a majority of the interviewed online participants. Again, this problem was exaeerbated by school 
budget euts that made it diffieult or impossible to use substitute teachers. Moreover, four of the 
non-teaehers interviewed noted that teaehers laeked time to attend or didn’t want to be away 
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from class. One teacher expressed frustration that he had to participate in online PD while 
managing his classroom: 

“The way it was presented to us was that we had to do it, it would be the last 30 
minutes of our last class, and somebody would come in and watch the classroom. 

Well, that was not happening and I could not pay attention to a webinar, dial in 
on my cell phone, listen and watch that computer monitor, and take care of my 
classroom. ” 

While perhaps not able to fully replace synchronous online PD, the coordinating 
difficulties could be partially addressed by making greater use of online resource libraries — 
something that both teachers and trainers felt could be advantageous. Most surveyed online 
participants indicated that they found access to online resources that they could use on their own 
time somewhat (25%) or very effective (45%) at engaging them in online PD. Aside from not 
having the coordinating difficulties associated with synchronous training, online resource 
libraries allow teachers to revisit ideas on an as-needed basis, and at their own pace. One teacher 
indicated that she was “looking for training that not only focused on new ideas and 2D* century 
curriculum, but that also offers the ability to periodically revisit ideas, rather than one that 
regurgitates everything in one day.” Another teacher talked about how she benefitted from 
repeating a PD session: 

“I think it is where the value of re-teaching comes in, because sometimes the 
same stuff that was presented the first time seems a little bit overwhelming. Then 
when you revisit it 4 months later all of a sudden more pieces fall into place, and 
as we go through it more and enact in the classroom, more and more pieces fall 
into place. ” 
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Lesson 2: Technical problems still emerge, even with adequate technology 

Most teachers and trainers felt that the available technology was adequate for the training 
activities. We found that technical problems did arise, however. A number of teachers reported 
audio problems that interfered with collaboration activities. Some technology problems were 
related to configuration or connectivity issues, while others were due to teachers’ (and 
sometimes trainers’) lack of knowledge or skill in using the technology. Some teachers also 
experienced problems logging into online sessions. Technical problems such as these distracted 
from the learning experience, caused wasted time, and caused frustration and anxiety. In some 
cases, teachers could not attend online PD or chose not to continue their participation in online 
training because of the technology problems. Although many teachers reported problems using 
the technology, not all felt that the problems impacted their ability to learn. 

“I don ’t think it [ technology problems] had any impact on my ability to learn. I’m 
not a really fast typer so sometimes I think I didn ’t get my point across fast 
enough where I was able to answer the questions or be a part of the conversation 
quick enough, but I definitely felt that I was able to learn. ” 

Nearly all of the trainers and coordinators believed that the technology available to 
teachers was adequate for the training activities. Teacher survey responses, however, were more 
variable. While the surveyed online participants “slightly agreed,” on average, that the 
technology provided by their school was adequate for participating in the online PD (mean = 4.3 
on a 6-point scale), nearly one -third moderately or strongly disagreed. Surveyed teachers were 
also asked about their experience and comfort with computers and technology, and results reveal 
that many teachers are uncomfortable with computer technology. Only 64% of teachers felt 
comfortable turning to an online discussion group when help is needed, 69% felt confident 
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troubleshooting internet problems, 65% felt that help is available when they experienee computer 
problems, and 29% reported having difficulties with most online programs they tried to use. 
However, only 13% felt that working with computers is very frustrating and 94% were confident 
in their ability to use a computer to learn. Online participants and nonparticipants did not differ 
much in their reported comfort with computers and technology, suggesting that comfort with 
technology did not affect participation decisions. 

Despite the adequacy of the technology available, technical problems were a major 
impediment to successful delivery of online PD. Problems with technology and the resulting 
wasted time was commonly cited, mentioned by nearly all of the teachers interviewed who 
participated in online PD. Only a few of the online participants indicated that technology 
problems deterred them from attending the online training. However, technology problems were 
more commonly cited as a deterrent to participation among interviewed nonparticipants. 

Audio problems were cited by about half of the interviewed online participants (52%) 
and included (a) difficulties with, or sometimes missing, headsets, (b) room configurations with 
phone lines on opposite ends of rooms from computers, (c) issues with school dial-out policies 
(some had to use cell phones), and (d) delay or lag in communication with the trainer that 
interfered with collaboration activities. Teachers ended up being able to communicate in most 
cases; however, some were forced to type or write rather than speak, and some could not hear 
what the trainer was saying. Configuration and computer equipment problems — as well as 
connectivity problems — were also commonly mentioned by interviewed online participants. 

Other problems were due to teachers’ (and sometimes trainers’) lack of knowledge or 
skill in using the technology, or problems logging into the online session. Training facilitators 
and coordinators interviewed mentioned similar technology problems, including connectivity and 
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login issues, room configuration and phone line issues, and headset or audio issues which 
hindered the start of the session. A common theme from the training facilitators and coordinators 
was that teachers lacked skill or comfort with the technology at first, but that skill and comfort 
improved over time, suggesting that with proper prior setup and training, the technology 
problems can be alleviated. 

The technical problems with the online PD distracted from the learning experience and 
wasted time. A few teachers commented that they were limited in how much they could cover 
due to time lost dealing with technical problems. Inevitably, the technical problems caused 
frustration and anxiety — and in a few cases were cited as reasons for discontinuing the online 
PD. One teacher remarked, “I only attended the sessions that were mandatory, because the 
technology problems were annoying and frustrating.” In an attempt to reduce these frustrations 
and anxiety, one of the training coordinators interviewed suggested that teachers would find 
great value in an initial “open house” to describe follow-up online training, set expectations, and 
practice how to interact with the technology. A few of the online participants interviewed 
indicated a desire to be trained on using the technology prior to the start of the PD (23%). 

Although many teachers reported technical problems, results indicated that they still 
found the training to be at least somewhat engaging. Most of the online participants interviewed 
indicated that the online PD was engaging or very engaging (55%); about half of the training 
facilitators and coordinators said that teachers seemed engaged in the online PD. However, in 
many cases, the technical problems impacted teachers’ ability to learn. One -third of online 
participants surveyed said that technology limited their ability to learn from online seminars and 
online resources they could use on their own time. It was also common for survey respondents to 
report that technology limited their learning from online collaborative tools (e.g., discussion 
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boards and blogs) and online tutorials. Despite teehnology problems, learning did oeeur; less 
than a third of interviewed online participants said that technology problems impacted their 
ability to learn from the PD (29%). Additionally, although some of the interviewed online 
participants indicated that the technology problems impacted their ability to learn from the online 
PD, technology problems were less likely to impact their opinion of online PD or their 
willingness to participate in more online PD. 

Lesson 3: Face-to-face PD is often preferred because of human contact and better engagement 
The online participants and nonparticipants both generally preferred face-to-face PD 
because of the human contact and feelings of engagement. Study groups were made available 
through online libraries and offered a low-cost and scalable alternative to face-to-face delivery, 
while still promoting human contact and engagement. 

Most of the teachers interviewed said that they preferred face-to-face PD (54%), and 68% 
of surveyed online participants agreed that online PD was less effective than face-to-face. 
Collaboration with other teachers and personal interaction with trainers was nearly unanimously 
cited as an advantage of face-to-face PD. Similarly, surveyed teachers who had participated in 
online PD “moderately agreed,” on average, that they would rather have had face-to-face training 
(mean = 5.1 on the 6-point scale). 

Low participation was mentioned by both teachers and trainers as an impediment to 
teacher interactions during online PD. One teacher remarked, “1 think that we would benefit 
more from more participants; they might think of something we didn’t.” One of the training 
facilitators concurred; 

“ ...one of the issues that worked against being able to get much collaboration 
going was that so few teachers actually attended ... I might have planned to 
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divide into groups and talk about something, but if you only have two people then 
that plan wasn ’t going to be very useful. ” 

Greater participation in online PD among teacher colleagues might improve feelings of 
engagement. Colleague support for attending online PD was mixed; 32% of online participants 
interviewed said that colleagues were supportive of online PD participation, while 26% said that 
colleague support was lacking or mixed. Teacher comments ranged from “Absolutely, I had a 
partner who attended the training in my school library with me” to “I would say with one or two 
exceptions, there was more ambivalence than encouragement.” 

Using materials downloaded from the online libraries provided the 9th grade teacher 
teams a low-cost and scalable alternative to face-to-face training that retained the human 
element. The study groups facilitated collaborative work in which teachers learned through 
discussing and working through the materials as a team. Nearly all of the respondents to the 
study group surveys agreed that the study group format is an effective way to learn new topics 
and that the study groups improved their team’s conversations and collaboration. Study group 
interviewees all reported that the study groups were useful in guiding discussions with 
colleagues. Five of the six people interviewed about their participation in study groups indicated 
that their implementation of the information learned has or will result in improved student 
learning. 

About one quarter of the interviewed online participants experienced problems with the 
online PD because of difficulty understanding the speaking of the training facilitators. While all 
facilitators and teachers spoke English, not all facilitators were from the same geographic region 
and/or ethnicity as the teachers, which may have led to some language barriers. Two of the 
online tutors for Mathematics Navigator were not native English speakers, which also 
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contributed to language barriers. These problems resulted in less engagement with the facilitators 
than would have otherwise been realized. 

Lesson 4: Teachers cited the convenience and value of online PD 

Although many teachers preferred face-to-face training, most reported value in the online 
training and that their teaching practices ehanged beeause of the training. Teaehers eited the 
convenience of online PD, indieated a desire for future online training, and acknowledged that 
online PD is the logical path for future training. 

“It [online training] definitely has a place. It can definitely help in bringing really 
good results, and I think it ’s something that is only going to get better. ” 

Teachers expressed enthusiasm about online PD. Nearly all online partieipants eited one 
or more of the following as key benefits of online PD: preservation of class time (16%), less 
travel (52%), more flexibility (35%), and cost savings (32%). Other commonly cited positive 
aspects included being able to communicate and share with others outside of their sehool or 
district (35%) and comfort of participating in their own surroundings and feeling that online 
training provided a safe, less intimidating environment (32%). One teaeher remarked, “During 
face-to-faee training some of them [teaehers] are shy or soared to speak up, but it seemed like 
over the web, people spoke up more. . . it seemed that people were a lot more open.” 

When refieoting on faoe-to-faoe and online PD related to their respective program, 
teachers reported the PD to be engaging and effeotive. On average, surveyed teaehers moderately 
agreed that “the PD actively engaged them in refieoting on their teaohing” and that the PD “will 
eventually result in greater student learning” (mean = 5.1 on the 6-point soale). Similarly, as 
mentioned previously, a majority of interviewed teachers found the online PD to be engaging 
(55%). Of the interviewed teaehers, most indicated an overall positive impression of both types 
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of PD. When reflecting upon their online training, most of the interviewees indicated that the PD 
influenced them positively, including providing them with new ways of teaching or new ways of 
learning (55%). 

The interviews also asked teachers about PD’s impact on certain teaching skills and 
practices. Online participants reflected on the impact of their online training while 
nonparticipants reflected on the impact of face-to-face PD. Interviewed online participants 
tended to indicate less impact of the PD on their skills and practices than did interviewed non- 
participants. A majority of the interviewed teachers indicated that the PD positively impacted 
their use of assessment data and student work to respond to student needs (52% and 74%, 
respectively, for online and face-to-face delivery), their ability to address student misconceptions 
(55% and 74%, respectively), the frequency or quality of collaborations with other teachers (55% 
and 68%, respectively), and their participation in professional learning communities (PLCs) 
within the school or district (52% and 58%, respectively). 

“When I had my training in person, we really didn ’t talk about or see how we 
could capture that [assessment data] and examine it either by standard or by 
students. The web training that I had... did look at that and gave us the 
information on how to do that. ” 

The level of the PD’s impact may not have been strong: On average, surveyed teachers 
only slightly agreed that their “confidence in teaching has increased as a result of the PD” (mean 
= 4.4 on the 6-point scale). However, as mentioned previously, five of the six people interviewed 
about their participation in study groups indicated that their implementation of the information 
learned has or will result in improved student learning. 
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While the study group training format is not eonsidered online PD, results suggest value 
in this alternative form of scalable training. Interviewees thought that the study groups improved 
their ability to use assessment results to understand and respond to students’ needs. In one 
teacher’s words, “The behavioral management (topics) gave you different insights into the 
student. . . We were able to find across the board with teachers, with the rating scale that we were 
having the same issues with the same student and it kind of gave us indicators to have a base 
point to decide what to do going forward with the student or how to better serve the student.” 
Another teacher said, “Some teachers don’t even consider sometimes why a student is acting the 
way they are. . . This kind of returns the personal aspect to teaching. . . that each individual has an 
individual situation.” 

Lesson 5: Careful system design, planning, and introduction can enhance online PD 

Most teachers thought that the PD could be improved and the impact optimized through 
better planning and preparation — addressing who, what, when, where, and why — and 
communicating these essential points to teachers in a timely fashion. Nearly half of surveyed 
teachers (44%) expressed dissatisfaction with the planning and organization of the online 
activities. Lack of planning and prior notification was noted by most interviewed online 
participants (68%), and nearly half of the nonparticipants said that they were not aware that 
online PD was being offered (53%). More online participants were contacted about the online 
PD via email (58%) than contacted in person (23%). 

“I found out about it [online PD] actually after it had taken place. ” 

Better coordination was the most common suggestion made by interviewed teachers for 
improving PD in their district, mentioned by over 40% of online participants and 
nonparticipants. Many teachers expressed a feeling that (a) adequate preparations had not been 



36 


made in ensuring teehnologieal fimetionality, or eritical informational materials were not 
available beforehand (e.g., manuals, student handbooks), and (b) as a result, they themselves 
were unprepared, didn't know what to expect, were confronted with obstacles or unanticipated 
problems, or went into training uncertain of why they were there. 

did not have any of the materials, so when the training was going on and the 
instructor told me to look at this or turn to this page, I could not do that... ” 

Lack of communication was mentioned by all nine of the training facilitators and 
coordinators interviewed; many teachers were notified of training just days before needing to 
attend. Less commonly, setting clear expectations was suggested by interviewed online 
participants as a way to improve online PD (16%). The study group interviews revealed that not 
all participants had the study group materials available for review prior to teacher team meetings. 
Teachers remarked that having the materials available would have made the meetings run more 
smoothly, and would have improved teacher buy-in to the study groups. 

Most interviewed teachers believed implementation of the program for which they 
received PD would result in improved student learning (63%). Thus, most teachers understood 
the value of their implementation efforts. Moreover, 85% of surveyed teachers agreed that they 
understood why they were asked to participate in PD. However, a theme that emerged from the 
teacher interviews was that some of the PD was not perceived as relevant. While nearly all 
teachers interviewed said that PD is important, 39% feel PD is important when relevant. A few 
teachers (11%) indicated that PD is usually a waste of time. A small number of teachers stated 
that while some of the PD they received was good, sometimes the material covered is something 
that they have already received training for or is self-explanatory. Teachers remarked that some 
of the training material didn’t require synchronous PD but could be self-directed (e.g., accessing 
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the online libraries). Interviewed online participants were less likely than nonparticipants to say 
that PD can be irrelevant or a waste of time (29% versus 68%). While we did not test a complete 
model for factors predicting participation in online PD, this finding suggests that perceived 
relevance may have affected decisions to participate in online PD. 

Some teachers remarked that PD (face-to-face and online) could be improved if 
leadership sought input from them with respect to their needs, the relevance of training, and the 
timing and scheduling of PD in their districts. Desire for teacher input was commonly mentioned 
by interviewed nonparticipants (42%), but less commonly cited by online PD participants (16%). 
One teacher stated it simply, “Just ask the people who are in the trenches a little bit more as to 
how it would be a better training for them (e.g., when it would be most convenient).” 

Lesson 6: There are common barriers to applying what is learned in PD 

In the surveys and interviews, teachers were asked if they were able to implement what 
they learned in PD and also to identify implementation barriers. Almost all teachers reported 
implementing what they learned in PD, to some extent. Common barriers included not enough 
support, lack of materials, lack of knowledge, learning the materials too late in the school year to 
have the chance to fully implement, and general lack of time. Another obstacle to 
implementation was poor student behavior and lack of classroom management. These issues 
affected teachers’ ability to implement programs as designed. One teacher remarked; 

“You have to focus a lot on behavior management, so it really impacted being 
able to move through the routine in the time that is recommended for each section 
of the workshop. ” 

For the teacher team study groups, a lack of time was the biggest obstacle to using what 
was learned. Interviewees mentioned this in response to multiple questions, and four of the six 
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interviewees mentioned time speeifieally as a barrier keeping them from applying what they 
learned in the study groups. In one teaeher’s words: 

“(There needs to be) enough time set aside to really go through the information 
as a team and make comparisons. It was really important to take the time that it 
takes to get the overall picture of the student data. ” 

Despite the implementation barriers, teachers were nearly unanimous in reporting that 
they had begun, and plan to continue, implementing the skills they learned during PD. This 
suggests that, despite the barriers discussed below, the PD was transferred to practice to some 
extent. 

Teachers expressed a sense that ongoing and frequent support was an important part of 
learning effective implementation strategies. Several sources of support were identified by 
teachers, including school or district facilitators/coordinators/coaches, school and district 
administrators, other teachers, and external training providers and program experts. Many 
teachers expressed reliance on and satisfaction with external program or topic experts. However, 
a small number of teachers commented that the level of support provided during the second year 
of implementation (2010-2011) declined drastically — noting that access to experts was better the 
previous year or that administration was more supportive the previous year. 

Nearly all teachers reported good support from school leaders for attending both online 
and face-to-face training. Most teachers also indicated that school administrators were committed 
to school improvement and full implementation of the Rigor & Readiness programs. However, a 
small number of the interviewed online participants indicated that administration was not 
supportive enough (19%), or that there was poor communication between administrators and 
teachers (10%). Also, surveyed teachers only “slightly agreed” that their supervisors allowed 
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them ample time to partieipate in online PD. Additionally, 29% indieated that they did not have 
the time to implement new strategies. One of the training eoordinators interviewed suggested that 
some teaehers reported having leadership support beeause they were told to attend the training 
(most online partieipants interviewed indieated that the online PD was required), but they did not 
reeeive the support they needed in terms of substitute teaehers to eover elass while they attended 
PD, materials needed for PD and implementation, or time to apply what was learned in PD to 
their everyday praetiee. 

Teachers want more time for peer interaction and review; however, our results indicate 
that teachers’ schedules provided little time with colleagues during school hours, and it was 
difficult for teachers to meet outside of scheduled hours because of personal commitments and 
union restrictions. Nearly half of the surveyed teachers indicated that they are not given adequate 
time to collaboratively plan (48%). Among surveyed teachers, 44% indicated that they do not 
visit colleagues’ classrooms to observe their teaching; interestingly, online participants were less 
likely to indicate that they visit colleagues’ classrooms (26%) than nonparticipants (70%). 
Colleague support for, and involvement with, the implementation of what teachers learned in PD 
was mixed; 48% of online participants and 63% of nonparticipants interviewed said that 
colleagues were supportive, while 35% of online participants and 26% of nonparticipants said 
that colleague support was lacking or mixed. 

Assuming that enough time is available for collaborative work, greater use of online 
libraries could help teachers apply PD to their work. In particular, those interviewed about using 
study groups as part of teacher teams had positive reactions to the collaboration it provided with 
other teachers. One teacher remarked “The study groups facilitated teachers sitting down and 
having dialogue.... sometimes I don’t always know the right questions to bring out the points 
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that we are needing to make and so I thought they just were great for faeilitating good 
discussions.” 

Teachers are very interested in collaborating with other teachers — not only from their 
schools — but with others from other schools and districts who are facing similar implementation 
challenges. A few teachers alluded to the potential for a wider professional learning community 
(i.e., nationwide) that could also function as a support network. A few teachers also said they 
benefitted from collaborations both with people they knew and those they did not. Having access 
to other educators allowed teachers to learn new instructional strategies, share ideas, gain new 
insight from questions asked by their peers, and gain perspective on topics of interest. One 
teacher commented “I liked that there were people that I did know to bounce ideas off and I liked 
the fact that there were people I didn’t know outside the district giving me a fresh look or a 
different view.” 

The study groups seem to be an effective format for fostering and facilitating teacher 
collaboration. Eighty percent of the survey respondents agreed that the study group format is an 
effective way to learn new topics and 85% thought the study groups improved their team’s 
conversations and collaboration. Interviewees all reported that the study groups were useful in 
guiding discussions with colleagues. 

A small number of the interviewed teachers indicated that lack of materials (e.g., program 
manuals, student handbooks) was a barrier to applying what they learned in PD to their practice. 
Specifically, not all districts were able to provide adequate materials for all students — due to 
either budget issues or coordinating problems. A small number of teachers also cited lack of 
knowledge of how to implement programs and lack of implementation time as barriers. 
Similarly, over a quarter of surveyed teachers expressed that they were not given the time they 
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needed to implement what they learned in PD (27%). Timeliness of the online PD was also 
mentioned by a few teaehers as a barrier to implementation. As one teaeher put it: 

“The training kind of came after I had already begun the program. So that was, 
you know, not as good a system as if I would’ve been trained prior to or at least 
did a couple trainings before I had to initiate a class. ” 

A particularly important and challenging barrier to implementation was student behavior 

and classroom management. Though less common, student behavior problems were cited by 

interviewed teachers, and were also mentioned by one of the training coordinators interviewed. 

Interestingly, classroom management was the most commonly mentioned topic for additional PD 

needed, requested by 39% of interviewed online participants and 37% of nonparticipants. From 

the interviews, we learned that teachers struggled with disruptions, poor student behavior, and 

willingness to participate. In some instances, school-wide disciplinary procedures did not always 

allow students to be suspended or removed from the classroom. Among surveyed teachers, 

nearly half reported that they do not have the tools needed to support students who are struggling 

with motivation and other behavioral issues that impact learning (49%). One teacher 

acknowledged struggling with understanding students’ backgrounds, saying “We don’t always 

understand about family dynamics, and we don’t all come from that kind of family.” 

Lesson 7: The school improvement climate can cause discontinuity, reducing expectations of 
online PD ’s impact 

The current atmosphere of school improvement present in the districts studied has some 
teachers feeling inundated with evolving mandates and multiple initiatives. The lack of 
continuity caused teachers to wonder whether the programs supported with PD would be around 
the following year. A few teachers expressed concern and frustration over the discontinuity, 
which may impede efforts to construct an ongoing collaborative culture of professionals 
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committed to program implementation. A small number of teachers indicated they were no 
longer teaching the same program the year following initial implementation. A few teachers 
eommented that “it’s just one program after the next” and that “this program will go away like 
the previous programs have.” Some teaehers, therefore, felt that it was wasteful to invest their 
time and energy to learn and implement a new program when it is just “going to go away.” 

Nearly one-third of the interviewed teachers mentioned that programs change frequently 
or that they are no longer teaehing or using the program (32%). A small number of teachers 
remarked that support for their program was better the previous year (4%) or that aceess to 
experts was better the previous year (10%). One of the program leaders interviewed also noticed 
teacher frustration with frequently changing improvement strategies: 

There were a lot of people who said to us, literally, “We doubt that you ’ll be back 
for the second year because we have done so many initiatives in this district and 
you ’ll just be a one-year wonder. ” 

Some teaehers expressed a desire for their schools/districts to “choose a program and 
stick with it.” Over 40% of surveyed teachers agreed that they often have difficulty choosing 
what to do in the classroom after hearing about so many “best” teaching practices. To combat the 
discontinuity problem and encourage teaehers to experiment with new strategies, a long-term 
commitment and extended schedule of expeeted results is needed. 

Perhaps due to the current accountability structure’s reliance on adequate yearly progress 
towards higher proficiency rates, teaehers are more inclined to utilize programs that they 
pereeived would not help students do well on state assessments. A small number of teachers 
expressed that although there are many good programs available, most seem to lack content 
relevant to individual state tests. As such, teachers feel following these programs with fidelity 
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results in a disservice for the students. One teacher remarked, “I felt like I didn’t teach them what 
they needed to know because I was following the program.” 

Despite the concerns with discontinuity, over 90% of surveyed teachers agreed that they 
had support and encouragement to implement new strategies from school leaders, and 67% of 
teachers agreed that the school improvement initiative (Rigor & Readiness) does not conflict 
with other job responsibilities. Some PD recipients indicated that their improvement efforts 
would be sustained and would lead to improved student performance. When asked whether 
implementation of what was learned from the study groups would result in improved student 
learning, a school principal remarked; 

“It has and it will continue to... My teachers understand there’s a big picture and 
how they fit into it. So, they know study groups are part of the plan, and helping 
them to understand data, how to use it, and helping them understand how to work 
together as a team. I absolutely think that we are going to see the results on our 
students’ performance. ” 

Summary and Recommendations 

This study examined teacher perceptions of online and face-to-face PD in four large 
school districts implementing an integrated school improvement initiative. The online PD of this 
study was designed to support core 9th grade mathematics curriculum, and aligned safety net 
programs in literacy and mathematics. Study groups, a training format that provides materials to 
facilitate face-to-face teacher team meetings, supported collaboration within teacher teams in 
middle and high schools. Survey and interview data on 101 individuals — including 51 teachers 
who participated in online PD or study groups — are used in this report. The study guided 
recommendations for practitioners implementing blended PD models tied to a conceptual map of 
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factors affecting successful delivery of the PD. Lessons learned from the study inelude lessons 
relating specifieally to online PD and others related to PD in general. Lessons learned specific to 
online PD include; 

1) Synchronous forms of online PD suffered from coordinating dijficulties, which may have 
contributed to low participation rates. Teaehers lacked common time for training and 
substitute teachers were not always available, eausing problems with classroom coverage. 

2) Teachers and trainers felt the technology available to them was adequate for the online 
training, but technical problems still emerged. The technical problems caused wasted 
time and impacted teaehers’ ability to learn. 

3) Most teachers preferred face-to-face over online PD — often because of its human contact 
and better engagement. Teaehers who partieipated in study groups as part of teaeher 
teams reported high levels of engagement and satisfaetion with interactions between team 
members. 

4) Teachers cited the convenience and value of online PD. While most teachers preferred 
face-to-faee training, most reported that the online training was valuable and convenient 
and that their teaching practices changed beeause of the training. 

Lessons learned related to PD in general include: 

5) Teacher buy-in and participation in PD could be enhanced with careful system design, 
planning, and introduction. Teaehers were not always aware that online training was 
being offered, and they pereeived a lack of coordination that affected their readiness for 
training. 

6) Common barriers to transfer what was learned in PD included lack of time and lack of 
ongoing support. Lack of materials, knowledge, and eollaboration opportunities were 
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other obstacles keeping teachers from fully implementing what they learned in training. 
Several teachers expressed a strong need for tools to improve classroom management. 

7) The current climate for school improvement caused discontinuity in approaches to 
raising student achievement, leading teachers to wonder whether the new programs 
supported with PD will persist. Some teachers expressed a desire for their school systems 
to “choose a program and stick with it.” 

Revisiting the Conceptual Map and Recommendations 

While the study focused on online PD for specific programs within a school improvement 
initiative, the findings have implications for the successful design and delivery of other online 
PD solutions for K-12 educators. The eight recommendations that follow are tied to a conceptual 
map of factors affecting successful delivery of PD. The conceptual map and recommendations 
were derived from our study, as well as from existing literature on factors affecting successful 
face-to-face, online, and blended PD models. The map and recommendations are intended to 
provide school system leaders and PD providers general guidance on implementing PD and 
understanding what the barriers are to PD having its desired impact. 

The conceptual map, presented in Figure 3, is organized by five categories of factors that 
are believed to affect, or be outcomes of, successful delivery of PD. Three of the five categories 
(contextual factors, logistics, and expectations/experiences) are considered supports for 
successful PD, while two categories (satisfaction and engagement and learning, transfer, and 
outcomes) are PD outcomes. The eight recommendations, discussed in greater detail below, are 
intended to improve the factors affecting successful PD. The conceptual map was modified from 
its original state presented in Table 1 and Figure 2. The modifications reflect insights we gained 
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from the study and judgments about the importance and interrelationships (directionality) of map 
components. The map’s components are described in Table 4. 


FIGURE 3. Revised Conceptual Map with Recommendations 
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TABLE 4 

Revised Conceptual Map Component Descriptions 


Component Name 

Component Description 

Supports and Barriers - Contextual Factors 

Step 1 of the implementation 
cycle: Set a unifying goal to 
work towards 

The extent to which a shared vision of school improvement, supported by district 
leadership, exists between school leaders and faculty and staff School leaders 
and teachers work towards shared goals that are measurable and attainable. 

School/classroom climate 

The qualities of the school and classroom environment that promote learning, as 
well as teachers’ ability and willingness to improve teaching. This includes the 
extent to which students feel safe and the extent to which the school and 
classroom are free of disruptions. 

Colleague support and 
involvement 

The degree that educator colleagues provide mutual support for participating in 
training, reflecting on teaching and how the training informs their practice, and 
transferring learning to practice. Colleague support and involvement also 
represents the degree that educators support each other’s teaching endeavors, in 
general, and implementation of new teaching strategies, in particular. 

Leadership support 

The aid and support provided to teachers by school leaders in pursuit of a 
common goal. Leadership support entails giving feedback, support, and 
encouragement through effective communication, ample time for learning and 
collaboration, and support of teachers’ experimentation with new strategies. 

Availability of training and 
classroom resources 

The extent to which training materials and classroom resources associated with 
the training are made available to teachers in a timely manner for use during 
both training and implementation. 

On-going support for training 
and implementation 

Teachers’ receipt of continued support from school leadership and training 
providers throughout the year for both training and implementation. This 
includes timely access to experts, resources for successful implementation, and 
adequate time for incorporating new ideas and collaborating with colleagues. 

Supports and Barriers - Logistics 

Convenience/time availability 

The extent to which scheduled training times offer convenience and flexible 
times for teachers to attend training. It also entails teachers being provided with 
ample time to attend without adversely affecting their teaching. 

Training technology and 
environment readiness 

The degree that teachers are familiar with the technology, tools, and the 
environment in which online PD will take place. Environment readiness also 
includes the school’s technology systems being adequately prepared for teachers 
to fully participate in online PD (e.g., system configuration, phone line and 
computer placement, functioning equipment). 

Knowledge of training topics 
and expectations 

The extent to which teachers have been adequately informed of upcoming 
training activities and expectations for those activities. This also includes 
understanding of the purpose and nature of the training activities and 
expectations for attendance and engagement. 

Supports and Barriers - Expectations/Experience 

Individual differences 
(motivation, teaching 
experience) 

Personal characteristics of teachers that affect their disposition towards PD. 
These include attitudes, personality factors, teaching experience, and personal 
circumstances that affect their ability to attend training. For example, one 
teacher may be motivated by professional responsibility, while another teacher 
may feel that all PD is a waste of time. 

Comfort and experience with 
technology 

Teachers’ level of comfort and experience with the technology needed for online 
training derived from their prior training or experience with the training tools. 

Expected value of training 

The level to which teachers believe that the planned training activities are 
worthwhile and will improve their teaching ability. 
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Satisfaction and Engagement 

Attendance and participation 

The degree to which teachers not only show up for training, but also actively 
participate in training activities. 

Engagement and retention 

The level to which teachers feel engaged in and are able to learn from the 
training activities, increasing the chances they want to return for more training 
sessions. 

Satisfaction and perceived 
value of training 

The extent to which teachers feel satisfied with the quality of training they 
received, as well as the extent that they believe the training was worthwhile and 
will improve their teaching. 

Learning, Transfer, and Ontcomes 

Change in attitudes, beliefs, 
knowledge 

The extent to which teachers’ attitudes, beliefs, and pedagogical content 
knowledge change as a result of the training they receive. 

Transfer to practice 
(implementation) 

The degree to which teachers are able to transfer what they learn in training to 
their teaching activities and responsibilities in and out of the classroom. 

Improved student learning 

The extent to which students’ learning gains improve based on PD-targeted 
strategies teachers have learned and implemented in their classrooms. 


Eight Recommendations for Practitioners 

1) Set a unifying goal to work towards — and stick with it 

Any school improvement effort, ineluding enhaneing PD offerings, should begin with 
identifying the goal of the sehool system. Many states and sehool systems have adopted eollege 
and career readiness as their end goal. This trend was evident in the development of the Common 
Core State Standards, whieh adopted the definition of eollege and eareer readiness as the 
aequisition of the knowledge and skills a student needs to enroll and sueeeed in eredit-bearing, 
first-year eourses at a postseeondary institution (sueh as a two- or four-year eollege, trade sehool, 
or teehnieal school) without the need for remediation (ACT, 2010). With a elear unifying goal to 
work towards, edueators and polieymakers ean adopt strategies that align to the goal and diseard 
strategies that do not. Teaehers will be more likely to benefit from PD if the training program is 
aligned tightly with other eomponents of the sehool system, ineluding standards, eurrieulum, and 
assessment — and if all eomponents work towards the common goal. Decisions of how to design 
or improve teaeher PD should be made with the unifying goal in mind. Setting a unifying goal, 
and stieking with it over multiple years, could have positive direet effeets on eomponents within 
the eoneeptual map, ineluding sehool/elassroom elimate, leadership support, and eolleague 
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support and involvement. It eould also improve teaehers’ motivation to partieipate in training, as 
well as their ability to antieipate training topics. 

While not an original focus of this study, one theme that emerged from the teacher 
interviews was that teachers are frustrated with programs that change frequently, sometimes after 
a single year (see lesson learned #7). They feel that programs cycle in and out, and they do not 
want to invest their time and effort into learning and trying to implement a new program only to 
abandon it the following year. This discontinuity problem might be alleviated by building tight 
alignment between school system components and PD offerings. With tight alignment, new 
strategies are less likely to be perceived as loosely connected programs because they fit in well 
with existing efforts. School system leaders can improve the impact of PD by sticking with well- 
conceived strategies that align to their end goal. Policymakers can also help by enacting laws and 
policies that allow more time for improved student achievement and meaningful accountability 
metrics that reflect students’ academic growth. These steps will reduce the temptation for school 
system leaders to abandon good strategies prematurely. 

2) Develop a continuous cycle of assessment and instruction of teacher needs 

“Teachers are students too” was a common refrain echoed within our study. Like their 
students, teachers could benefit from a continuous cycle of assessment of their areas of need and 
delivery of PD to meet those needs. Developing a continuous cycle of assessment and instruction 
of teacher needs could have positive effects on components within the conceptual map, including 
knowledge of training topics and expectations, expected value of training, and perceived 
satisfaction and value of training. 

Assessment could take several forms, including self-help inventory or teacher-specified 
needs, classroom observation, content and pedagogical knowledge assessment (Goldschmidt & 
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Phelps, 2010), student survey (The Bill and Melinda Gates Foundation, 2010b), value-added 
statistical analysis using student assessment data, and peer review. In response to this assessment 
challenge, a large-scale research study of measures of teaching effectiveness is seeking to 
develop fair and reliable measures of effective teaching that can be used to help identify 
teachers’ needs (The Bill and Melinda Gates Foundation, 2010b). 

Assessing what teachers need ensures that teachers help determine which PD topics they 
receive, which our study suggested is important. Teachers suggested that teacher buy-in, 
satisfaction, and participation might improve if teachers are part of the PD design and planning 
process. Prior research and policy work supports this finding, and suggests that teachers are more 
likely to support improvement efforts if they are consulted during system design (Corcoran, 
1995). 

While we did not test a complete model for factors predicting participation in online PD, 
our results suggest that perceived relevance may have affected decisions to participate in online 
PD. This underscores the need for targeting PD to individual teachers’ needs — in the same way 
that teachers strive to target instruction towards their students’ unique needs. Because every 
teacher’s assessed needs are likely to be different, PD offerings should be somewhat flexible or 
modularized. Our study revealed that teachers often view PD as a waste of time; this design 
element should help teachers see relevance and value in their PD. 

The continuous cycle of assessment should inform individual teachers’ needs, as well as 
aggregate needs for the school system. Aggregated measures of student academic achievement, 
psychosocial factors, demographics (e.g., English language learner population), and other 
measures (e.g., absenteeism rates, number of students with special learning needs) can also help 
bring a school system’s need for certain PD topics into focus. 
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3) Develop flexible, relevant models of quality PD 

Recent studies (Hidalgo, 2010; U.S. Department of Education, 2010) suggest that models 
that blend face-to-face and online delivery may be most effective and economical. There is a 
great need for more ongoing, targeted training that helps teachers address obstacles as they meet 
them during the school year — and online PD makes it easier to deliver just-in-time training. 
Developing flexible and relevant models of quality PD might have positive effects on ongoing 
support for training and implementation, convenience and time availability, and training 
technology and environment readiness. (See Table 4 for descriptions of these map components.) 

An important design consideration is to determine what PD should be delivered face-to- 
face and what can be delivered online. Careful analysis of the PD curriculum and learning 
objectives, along with research of online learning theories, can inform these decisions. For PD 
that need not be face-to-face, analysis must also address whether it needs to be synchronous. Our 
study suggests that teachers want to be more involved in PD design. One way that teachers can 
help design PD is to analyze which delivery methods (e.g., face-to-face, online seminar, online 
tutorial, online libraries, or study groups for teacher teams) are most appropriate for specific 
learning objectives. 

Our study demonstrated that synchronous forms of online PD can have significant 
coordination and technical problems (lessons learned #1, 2). Online libraries and other non- 
synchronous forms of online PD are less likely to have these problems. Moreover, recorded 
training sessions (either online or face-to-face) offer teachers the opportunity to view missed 
training sessions or to review particular sessions of interest. Positive training experiences can 
also be captured through threaded discussions and sharing of student work. One day’s 
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synchronous experiences can become part of tomorrow’s online libraries. The online libraries 
faeilitate review and new staff training — and eould make these activities more affordable. 

4) Build plans for online PD into master sehool schedules 

We learned in this study that scheduling and eoordinating online PD ean be more diffieult 
than seheduling face-to-faee PD (lesson learned #1). This was mainly due to faee-to-face training 
being seheduled further in advance, but was also due to diffieulties in finding common planning 
times aeross sehools for synehronous online PD. Building plans for online PD into master school 
schedules may have positive effeets on eoneeptual map eomponents such as colleague support 
and involvement, eonvenienee/time availability, and attendance and participation. 

In order for teaehers to benefit from synehronous forms of online PD, they must have 
time to prepare and attend. If online PD is offered during sehool hours outside of teachers’ 
planning times, substitute teaehers are needed for elassroom coverage. Sehool system leaders ean 
improve synehronous online PD offerings by building plans into master sehool sehedules and 
synching up common planning times for same-subject teachers aeross sehools. This would afford 
teachers the opportunity to participate at the same time as their eolleagues within and aeross 
sehools. In our study, teaehers felt that the online PD lacked human engagement (lesson learned 
#3), in some eases due to low participation rates; our findings suggest that greater participation in 
online training by groups of teaehers would improve engagement and satisfaetion. 

5) Enact aceountability for participation and eommunication 

The online PD environment is different from a faee-to-faee PD session. Teaehers may 
feel comfortable expressing opinions that they would not express in a face-to-face session. 
Unfortunately, the perceived anonymity of the online environment might also deerease feelings 
of aecountability, resulting in less engagement and partieipation. An accountability mechani s m 
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should be in place to ensure that teachers not only attend training, but participate and engage at a 
high level. Enacting accountability for participation in (and communication of) training may 
have positive effects on knowledge of training topics and expectations, expected value of 
training, attendance and participation, and engagement and retention. (See Table 4 for 
descriptions of these map components.) 

One way to encourage teachers to attend online PD is to provide incentives, such as 
continuing education or professional learning credits. Most teachers believe that PD is very 
important; however, their time is limited by teaching, planning, and other professional duties. If 
online PD cuts into that time, they might not place a high priority on it relative to their other, 
more immediate, responsibilities. Recognizing teachers for participation may be one way to 
make online PD a higher priority. 

As evidenced in our study, clear lines of communication are needed for successful 
delivery of online PD (lesson learned #5). Any PD system design should specify who is 
responsible for communicating the “who, what, where, when, and why” information to teachers. 
Ensuring that teachers know what to expect will help them get the most out of PD. They need to 
be given information about the purpose and nature of the PD, including expectations for 
engagement, so that they can be prepared to participate. 

6) Equip teachers to engage in online PD 

The PD system design must ensure that teachers have the tools necessary for full 
participation, including the proper computer setup, headphones, digital tablets, or other necessary 
equipment, as well as any training materials needed. Ensuring that the computer system is 
working properly is not sufficient (lesson learned #2); teachers need to practice using the 
technology prior to engaging in online PD. Teachers need to be familiar and comfortable with 
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the technology to minimize frustration and anxiety, allowing them to fully engage in the learning 
experience. Equipping teachers to engage in online PD addresses conceptual map components 
such as availability of training and classroom resources, ongoing support for training and 
implementation, training technology and environment readiness, and comfort and experience 
with technology. 

Early face-to-face PD sessions could include training to familiarize teachers with the 
tools they will be using in the online PD. This would help teachers and PD coordinators 
anticipate technology problems and correct them before online training commences. 
Additionally, beginning PD with face-to-face sessions before transitioning to online follow-ups 
could help build the community of learning that continues when online sessions commence. This 
could help address teachers’ expressed desire for personal interaction and collaboration, 
especially if it results in higher levels of participation in the follow-up training. 

Recommendations for improving engagement in online PD can also be drawn from 
standards for K-12 online teaching. The National Standards for Quality Online Teaching 
contains guidelines for the design and delivery of online training (North American Council for 
Online Teaming, 2010). The standards refer to “teachers” and “students.” In the case of online 
PD, “teachers” are the providers of the online PD and “students” are the teachers receiving the 
online PD. Standards C, D, and M are relevant to providing an environment in which teachers 
can participate in and learn from the PD. Standard C states “The teacher plans, designs, and 
incorporates strategies to encourage active learning, interaction, participation, and collaboration 
in the online environment,” and Standard D states “The teacher provides online leadership in a 
manner that promotes student success through regular feedback, prompt response, and clear 
expectations.” Standard M states “The teacher arranges media and content to help students and 
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teachers transfer knowledge most effectively in the online environment.” In other words, the 
online PD should be designed in such a way as to take advantage of the unique benefits provided 
by the online medium and to facilitate teachers’ ability to engage in and learn from the PD. 

7) Deliver targeted training throughout the year 

Teachers in our study indicated a desire for having more periodic training throughout the 
year rather than concentrating all training during the summer or at the start of the school year. 
Continuous training allows teachers to ask questions as they arise and share experiences and 
advice with one another. A blended solution may be a cost effective way to provide this just-in- 
time training. Early face-to-face PD sessions can be used for community building which can be 
sustained throughout the school year in online PD sessions. Delivering targeted training 
throughout the year addresses conceptual map components such as expected value of training, 
engagement and retention, and satisfaction and perceived value of training. 

8) Equip teachers to implement what they learn in PD 

Eor PD to have its desired impact, teachers must have the resources and ongoing support 
needed to implement what they learned. Common barriers to implementation included lack of 
time, lack of materials, lack of knowledge, and learning the materials too late in the school year 
to have the chance to fully implement (lesson learned #6). Equipping teachers to implement what 
they learn in PD enhances colleague support and involvement, ongoing support for training and 
implementation, and transfer to practice. (See Table 4 for descriptions of these map 
components.) 

Another obstacle to implementation was poor student behavior and lack of classroom 
management. Interestingly, classroom management was the most commonly mentioned topic for 
additional PD needed, requested by nearly 40% of teachers interviewed. Improving student 
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behavior and classroom management could potentially improve the transfer of PD-targeted 
knowledge and practices to classroom implementation. Classroom interruptions such as those 
caused by students talking off-task, students or others walking into the classroom unexpectedly, 
and students missing materials may be hindering teachers’ ability to improve instruction. 
Disruptions — especially those caused by students — can create a great deal of anxiety for teachers 
and compromise daily lesson structures, requiring teachers to develop adaptive strategies that 
may not be consistent with improvement strategies. In our study, some of the PD-targeted 
instructional strategies involved group activities, which may actually increase the likelihood of 
distractions in the classroom, thus undermining their purpose (Kennedy, 2005). 

Making more time for teacher collaboration can also help teachers implement what they 
learn in PD. With colleagues to bounce ideas off of and to share successes and failures with, 
teachers may be less likely to feel alone in their struggle to apply new ideas in their classrooms. 
Other studies also point to a lack of administrative support and adequate time for planning and 
implementation as a common problem faced by teachers. It is estimated that 50% of teachers 
leave the profession permanently after only five years, and a frequently cited reason for this 
departure is lack of administrative support (Lewis, 2010). Teachers in high performing countries, 
on average, are provided with more time to plan and collaborate. For example, in many high- 
performing European and Asian countries, teachers spend about 60% of their working time 
teaching students, and spend the remaining 15 to 20 hours per week engaging in other activities 
such as planning, meeting with students and parents, and collaborating with other teachers. In the 
U.S., teachers typically spend about 80% of their working time teaching students, and have about 
3 to 5 hours a week for planning, which tends to be less collaborative (Wei, Darling-Hammond, 
Andree, Richardson, & Orphanos, 2009). 
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Also, by following the first recommendation (“set a unifying goal to work towards — and 
stick with it”), school system leaders would signal to teachers that they are supportive of their 
implementation efforts and that teachers are expected to improve and adapt their implementation 
over a long period of time — not a matter of weeks or months. Teachers might then be more likely 
to apply what they learned in training, knowing that their efforts will not be wasted due to new 
replacement initiatives. 
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Appendix A: Survey and Interview Results 

Results are organized by the eoneeptual map’s seetions (eontextual faetors, logisties, 
expeetations and experienees, satisfaetion and engagement, and learning, transfer, and 
outeomes). Eaeh seetion eontains results from three data sourees: non-teaeher interviews, teaeher 
interviews, and teaeher surveys. The abbreviations in the tables are as follows: “OP” refers to 
online partieipants and “ONP” refers to nonpartieipants. Unless noted, the numbers in the tables 
are eounts of the number of individuals endorsing a partieular statement. 

When summarizing interview results, we often eompare the pereentage of teaehers who 
endorsed a eertain theme. The denominators used for ealeulating the pereentages are 31 for the 
online partieipants and 19 for the online nonpartieipants, refleeting the number of teaehers who 
reported online partieipation at the start of the interview. 

Contextual Factors 

Table A1 eontains results from interviews of training faeilitators and eoordinators 
regarding whether the training was already part of the distriet plans prior to the start of the sehool 
year. One partieipant indieated that it was in the original plans, while three indieated that it was 
not in the original plans, two indieated that a eommunieation plan was laeking, and one said that 
teaehers were not always aware that the training was going on. 


Table Al. District Plans (non-teacher interview) N 

Not in original plans 3 

In original plans 1 

Communication plan lacking 2 

T eachers not always aware 1 


Table A2 eontains eoded eomments from teaeher interviews regarding pereeived 
diseontinuity in sehool improvement programs from year to year. Some teaehers indieated that at 
their sehool, programs are introdueed one year only to be abandoned within a year or so. 
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Similarly, a few teachers indicated that support for their program was better the previous year. 
Overall, about one -third of the teachers interviewed mentioned one or more of these issues. 


Table A2. Discontinuity (teacher interview) 

OP 

ONP 

Total 

Programs change frequently 

4 

3 

7 

No longer teaching program 

3 

2 

5 

Administrative support better last year 

0 

2 

2 

Access to experts better last year 

3 

2 

5 

One or more discontinuity issues 

9 

7 

16 

Table A3 contains results from the teacher survey indicating 

the extent to which teachers 

feel able to choose between multiple teaching practices. Teachers 

appear to have some slight 

difficulty choosing what to do in the classroom after hearing about many 

“best” 

teaching 

practices. 




Table A3. 1 often have difficulty choosing what to do in my classroom after 
hearing about so many “best” teaching practices, (teacher survey) 

OP 

ONP 

Total 

Strongly Disagree 

4 

6 

10 

Moderately Disagree 

5 

4 

9 

Slightly Disagree 

6 

4 

10 

Slightly Agree 

2 

7 

9 

Moderately Agree 

4 

6 

10 

Strongly Agree 

0 

2 

2 

Not Applicable 

0 

1 

1 

Mean 

2.9 

3.3 

3.1 

Table A4 contains results from the teacher survey regarding alignment between 

PD and 

school improvement efforts. Overall, teachers indicated slight to moderate disagreement with the 

statement that the PD activities are not aligned with other school improvement 

efforts (mean = 

2.5), meaning that overall, they believe they are at least somewhat aligned. 



Table A4. The PD activities don’t seem aligned with our other school 
improvement efforts, (teacher survey) 

OP 

ONP 

Total 

Strongly Disagree 

6 

9 

15 

Moderately Disagree 

5 

9 

14 

Slightly Disagree 

7 

4 

11 

Slightly Agree 

2 

4 

6 

Moderately Agree 

0 

5 

5 

Strongly Agree 

1 

0 

1 

Mean 

2.4 

2.6 

2.5 


Table A5 contains results from the teacher survey indicating the extent to which teachers 


perceive an alignment between Rigor & Readiness and their other job responsibilities. Overall, 
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teachers slightly to moderately agreed that Rigor & Readiness did not conflict with their other 
job responsibilities (mean = 4.4). 


Table A5. Implementation of Rigor & Readiness does not conflict with my other 
job responsibilities (teacher survey) 

OP 

ONP 

Total 

Strongly Disagree 

3 

1 

4 

Moderately Disagree 

2 

2 

4 

Slightly Disagree 

2 

6 

8 

Slightly Agree 

2 

3 

5 

Moderately Agree 

4 

6 

10 

Strongly Agree 

7 

11 

18 

Not Applicable 

1 

0 

1 

Mean 

4.2 

4.5 

4.4 


Table A6 contains results from teacher interviews regarding perceived administrator 
support for Rigor & Readiness. Most teachers indicated that administrators were committed to 
school improvement and full implementation of the Rigor & Readiness programs, with 58% and 
68%, respectively, of online participants and nonparticipants indicating that they were very 
supportive. However, 19% of the online participants indicated that administration was not 
supportive enough, and 10% indicated that there was poor communication between 
administration and teachers. 


Table A6. To what extent were administrators committed to school 
improvement and full implementation of the Rigor & Readiness programs? 
(teacher interview) 

OP 

ONP 

Total 

Provided time to implement 

0 

1 

1 

Provided resources 

1 

0 

1 

Very supportive 

18 

13 

31 

Skeptical 

0 

1 

1 

Not involved enough 

6 

0 

6 

Poor communication 

3 

0 

3 

Lacked resources/support 

4 

0 

4 

More supportive last year 

0 

2 

2 


Table A7 contains results from teacher interviews regarding the extent to which 
colleagues were supportive of and involved in implementation of the program for which they 
received PD. Nonparticipants were more likely to indicate that their colleagues were supportive 
(63%) as compared to online participants (48%). Some online participants indicated that their 
colleagues were skeptical of the program (23%). 



62 


Table A7. To what extent were your colleagues supportive of and involved in 

OP 

ONP 

Total 

the implementation of 

Supportive 

? (teacher interview) 

15 

12 

27 

Not supportive 


3 

0 

3 

N/A or don’t recall 


2 

0 

2 

Skeptical 


7 

3 

10 

Split 


1 

2 

3 


Table A8 contains results from interviews with the 9 training facilitators and coordinators 
regarding their perceptions of leadership support to attend online training. Failure to 
communicate was the most common response (n=4). Participants also indicated that leadership 
was lacking (n=2) or that support varied (n=2). Two participants did indicate that leadership was 
supportive. 


Table A8. Leadership Support (non-teacher N 

interview) 

Supportive of initiative 2 

Failed to communicate 4 

Leadership lacking 2 

Support varied 2 


Interviewed teachers were also asked about leadership support, and the results are 
presented in Table A9. Overall, teachers reported good leadership support, with 81% and 89%, 
respectively, of online participants and nonparticipants reporting good leadership support. Four 
online participants reported a lack of leadership support for attending online training, and three 
nonparticipants reported a lack of leadership support for attending online training. None of the 


teachers who received face-to-face PD indicated a lack of support for attending face-to-face PD. 


----- - — - — p. p. 

Table A9. Was the principal or other school leadership supportive of you 

- o 

OP 

ONP 


attending training? (teacher interview) 

Online 

Online 

F2F 


PD 

PD 

PD 

Good leadership support 

25 

3 

17 

Lacked leadership support 

4 

3 

0 

N/A or don’t recall 

1 

3 

0 


Table AlO contains results from teacher interviews regarding whether training was 
required or optional. Most online participants indicated that it was required or strongly 
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encouraged (71%). None of the nonparticipants indicated that the face-to-face PD was optional 
or that the online PD was required. 


Table AlO. Was training required or optional? (teacher interview) 

OP 

Online 

PD 

ONP 

Online 

PD 

F2F 

PD 

Required 

18 

0 

12 

Optional 

5 

5 

0 

Not sure or don’t recall 

3 

1 

1 

Recommended/ encouraged 

4 

0 

3 


Table All contains results from teacher surveys regarding whether online participants 
were provided with enough time to participate in online PD. Overall, online participants 
indicated slight agreement that their supervisors allowed them ample time to participate in online 
PD (mean = 4.3). 


Table All. My supervisor allowed ample time for me to participate in web- OP 

based PD. (teacher survey) 

Strongly Disagree 1 

Moderately Disagree 3 

Slightly Disagree 2 

Slightly Agree 4 

Moderately Agree 4 

Strongly Agree 6 

Mean 4.3 


Table A12 contains results from teacher surveys regarding whether teachers were given 
adequate time to collaboratively plan. Online participants tended to slightly to moderately 
disagree that they were given enough time (mean = 2.8), whereas nonparticipants tended to 
slightly agree that they were given enough time (mean = 4.2). 


Table A12. Teachers in my department are given adequate time to 
collaboratively plan, (teacher survey) 

OP 

ONP 

Total 

Strongly Disagree 

7 

3 

10 

Moderately Disagree 

5 

5 

10 

Slightly Disagree 

2 

3 

5 

Slightly Agree 

2 

2 

4 

Moderately Agree 

2 

6 

8 

Strongly Agree 

3 

11 

14 

Not Applicable 

0 

1 

1 

Mean 

2.8 

4.2 

3.6 
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Table A13 contains results of the teacher survey regarding the perceived adequacy of the 
technology in the schools. Overall, the online participants indicated slight agreement that the 
technology provided by their school was adequate for participating in online PD (mean = 4.3). 


Table A13. The technology provided by my school was adequate for my OP 

participation in web-based training, (teacher survey) 

Strongly Disagree 3 

Moderately Disagree 1 

Slightly Disagree 2 

Slightly Agree 2 

Moderately Agree 6 

Strongly Agree 6 

Mean 4.3 


Table A14 contains results from the teacher survey regarding the extent to which teachers 
believe they have the tools they need to identify struggling students. Overall, teachers slightly 
agreed that they have the tools they need to identify struggling students (mean = 3.8). Online 
participants were slightly less likely to agree with the statement (mean = 3.7) than 
nonparticipants (mean = 3.9). 


Table A14. 1 have tbe tools I need to identify students wbo are struggling with 
motivation or other behavioral issues that impact learning, (teacher survey) 

OP 

ONP 

Total 

Strongly Disagree 

2 

5 

7 

Moderately Disagree 

4 

3 

7 

Slightly Disagree 

3 

4 

7 

Slightly Agree 

6 

3 

9 

Moderately Agree 

2 

9 

11 

Strongly Agree 

4 

6 

10 

Not Applicable 

0 

1 

1 

Mean 

3.7 

3.9 

3.8 


Table A15 contains results from the teacher survey regarding the extent to which teachers 
believe they have the tools they need to support struggling students. Overall, teachers were split 
with respect to the extent to which they have the tools they need to support struggling students 
(mean = 3.5). Online participants slightly disagreed (mean = 3.3), whereas nonparticipants were 
split (mean = 3.5). 
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Table A15. 1 have the tools I need to support students who are struggling with 
motivation and other behavioral issues that impact learning, (teacher survey) 

OP 

ONP 

Total 

Strongly Disagree 

1 

8 

9 

Moderately Disagree 

6 

1 

7 

Slightly Disagree 

6 

3 

9 

Slightly Agree 

3 

6 

9 

Moderately Agree 

3 

9 

12 

Strongly Agree 

2 

3 

5 

Not Applicable 

0 

1 

1 

Mean 

3.3 

3.5 

3.5 


Table A16 contains results from teacher interviews regarding access to experts outside of 
PD. Overall, teachers indicated that they had good access to program or topic experts outside of 
PD, with 68% and 79%, respectively, of the online participants and nonparticipants indicating 
that they had good access. Three online participants and two nonparticipants indicated that their 
access to experts was better the previous year. 


Table A16. To what extent did you have access to program or topic experts 
outside of PD? (teacher interview) 

OP 

ONP 

Total 

Good access 

21 

15 

36 

No access 

3 

0 

3 

Access was better last year 

3 

2 

5 

Limited access 

5 

1 

6 


Table A17 contains results from the teacher survey regarding access to experts outside of 
PD. Overall, teachers were slightly to moderately satisfied with their access to experts outside of 
the face-to-face PD (mean = 4.5), and slightly satisfied with their access to experts outside of the 
online PD (mean = 3.8). Online participants were moderately satisfied with expert access outside 


of face-to-face PD (mean = 4.8), whereas nonparticipants were slightly satisfied with expert 
access outside of face-to-face PD (mean = 4.3). 


Table A17. Please rate your satisfaction with access Online 


to experts outside of PD (teacher survey) PD 

OP 

Very Dissatisfied 4 

Moderately Dissatisfied 0 

Slightly Dissatisfied 0 

Slightly Satisfied 2 

Moderately Satisfied 2 

Very Satisfied 4 

Not Applicable 7 

Mean 3.8 


F2F PD 

OP ONP Total 

1 2 3 

1 2 3 

2 2 4 

1 7 8 

6 5 11 

8 8 16 

2 4 6 

4.8 4.3 4.5 
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Table A18 contains results from the teacher survey regarding the extent to which teachers 
have administrative support to implement new strategies. Overall, teachers moderately agreed 
that they have support to implement new strategies (mean = 4.9). 


Table A18. 1 have support and encouragement to implement new strategies 
from administrators at my school, (teacher survey) 

OP 

ONP 

Total 

Strongly Disagree 

1 

2 

3 

Moderately Disagree 

0 

0 

0 

Slightly Disagree 

0 

1 

1 

Slightly Agree 

6 

5 

11 

Moderately Agree 

6 

9 

15 

Strongly Agree 

7 

13 

20 

Not Applicable 

1 

1 

2 

Mean 

4.9 

4.9 

4.9 

Table A19 contains results from the teacher survey regarding the extent to which teachers 

were given adequate time to implement Rigor & Readiness. Overall, teachers slightly agreed that 

they were given the time they needed to implement Rigor & Readiness (mean 
participants were more likely to disagree (mean = 3.7) than nonparticipants (mean 

= 4.2). Online 
= 4.5). 

Table A19. 1 am given the time I need in order to implement Rigor & Readiness 
programs, (teacher survey) 

OP 

ONP 

Total 

Strongly Disagree 

5 

1 

5 

Moderately Disagree 

2 

2 

2 

Slightly Disagree 

7 

3 

7 

Slightly Agree 

11 

6 

11 

Moderately Agree 

18 

12 

18 

Strongly Agree 

8 

6 

8 

Not Applicable 

0 

0 

0 

Mean 

3.7 

4.5 

4.2 


Table A20 contains results from teacher interviews regarding implementation barriers. 


Common themes included lacking materials (16% of both groups), student behavior issues (10% 
and 16%, respectively, of online participants and nonparticipants), lacking time (6% and 16%, 
respectively, of online participants and nonparticipants), and no longer teaching the program 
(10% and 11%, respectively, of online participants and nonparticipants). 
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Table A20. Are there problems or barriers that have kept you from applying 
what you learned in training to your teaching duties? (teacher interview) 

Lacking time 
Lacking knowledge 
Lacking materials 
No longer teaching program 
Lacking support 
Student behavior issues 


OP ONP Total 

2 3 5 

3 1 4 

5 3 8 

3 2 5 

3 0 3 

3 3 6 


Nonparticipants surveyed were asked why they did not participate in online PD, and the 


results are presented in Table A21. The most eommon response was that they were not told about 


the training (55%), followed by they did not have enough time or had seheduling conflicts 


(23%), and their supervisor did not ask them to participate (13%). 


Table A21. If you did not participate in web-based PD, what ONP 

are the reason(s) why not? (Check all that apply.) (teacher 

survey) 

I wasn’t told about the web-based training 17 

Not enough time / scheduling conflicts 7 

I don’t need training 1 

Lack of resources or limited technology 2 

My supervisor did not ask me to participate 4 

I did not have enough prior notice to attend 3 

Other 1 


Table A22 contains results of the teaeher survey regarding whether teaehers visit 


colleagues’ elassrooms to observe their teaching. Online partieipants were less likely to indicate 


that they visit eolleagues’ elassrooms (26%) than nonpartieipants (70%). 


Table A22. 1 visit colleagues’ classrooms to observe their teaching, (teacher 
survey) 

OP 

ONP 

Total 

Strongly Disagree 

8 

5 

13 

Moderately Disagree 

3 

2 

5 

Slightly Disagree 

3 

2 

5 

Slightly Agree 

5 

9 

14 

Moderately Agree 

0 

3 

3 

Strongly Agree 

0 

9 

9 

Not Applicable 

2 

1 

3 

Mean 

2.3 

4.0 

3.3 


Table A23 eontains results of the teaeher survey regarding whether teaehers meet with 
others who teaeh the same grade-level eourse to discuss student work and assessment. Overall, 
teaehers slightly agreed (mean = 3.9). There were substantial differenees between online 
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participants and nonparticipants; online participants indicated slight disagreement that they 


discussed student work and assessment with other teachers 

(mean = 

= 3.0), 

whereas 

nonpartieipants indieated moderate agreement (mean = 4.5). 

Table A23. 1 meet with teachers who teach the same grade-level course 
regularly to discuss studeut work aud assessmeut items, (teacher survey) 

OP 

ONP 

Total 

Strongly Disagree 

4 

1 

5 

Moderately Disagree 

4 

2 

6 

Slightly Disagree 

3 

3 

6 

Slightly Agree 

7 

7 

14 

Moderately Agree 

3 

7 

10 

Strongly Agree 

0 

8 

8 

Not Applicable 

0 

3 

3 

Mean 

3.0 

4.5 

3.9 

Table A24 contains results from the teaeher survey regarding the 

extent 

to whieh 

principals regularly share information on school progress. Online partieipants, on 

average. 

slightly agreed that their prineipal regularly shared information (mean 
nonpartieipants, on average, slightly to moderately agreed (mean = 4.6). 

= 4.0), 

whereas 

Table A24. Our priucipal regularly shares iuformatiou ou school progress, 
(teacher survey) 

OP 

ONP 

Total 

Strongly Disagree 

3 

2 

5 

Moderately Disagree 

1 

4 

5 

Slightly Disagree 

3 

0 

3 

Slightly Agree 

4 

4 

8 

Moderately Agree 

7 

7 

14 

Strongly Agree 

3 

13 

16 

Not Applicable 

0 

0 

0 

Mean 

4.0 

4.6 

4.4 


Table A25 eontains results from the teaeher survey regarding the extent to whieh 


leadership faeilitates two-way eommunieation between teaehers and leaders. Overall, teaehers 
slightly agreed that sehool leaders faeilitate two-way eommunieation (mean = 3.9). 
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Table A25. Leadership in our school facilitates two-way communication, 
providing information and receiving feedback from teachers and others, 
(teacher survey) 

OP 

ONP 

Total 

Strongly Disagree 

3 

5 

8 

Moderately Disagree 

1 

3 

4 

Slightly Disagree 

5 

2 

7 

Slightly Agree 

2 

7 

9 

Moderately Agree 

8 

7 

15 

Strongly Agree 

2 

7 

9 

Not Applicable 

0 

0 

0 

Mean 

3.8 

3.9 

3.9 


Table A26 contains results from teacher interviews regarding the extent to which teachers 


believed that the program they were implementing would work well for their students. Overall, 
more online participants indicated that they thought the program would be a good fit for their 
students (52%) than did nonparticipants (21%). Several nonparticipants indicated that they were 
skeptical of the fit of the program for their students (37%). 


Table A26. Prior to implementation, did it seem like 
well for your students? (teacher interview) 

would work 

OP 

ONP 

Total 

Good fit 


16 

4 

20 

Not good fit 


4 

1 

5 

No expectation 


4 

4 

8 

New lower program 


0 

2 

2 

Skeptical 


0 

7 

7 


Logistics 


Table A27 contains a summary of themes relating to planning and scheduling difficulties 
mentioned during interviews with training facilitators and coordinators, including problems with 
availability of substitute teachers (56%) and scheduling issues (56%). A dominant theme 
concerned communication issues, and was mentioned by every participant of the non-teacher 
interviews. These communication issues tended to be related to lack of communication from 
districts to principals to teachers, and were mostly related to problems scheduling teachers and 
teachers not knowing what to expect of the training. 
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Table All. Planning Themes (non-teacher N 

interview) 

Substitute teachers 5 

Communication issues 9 

Scheduling issues 5 


Tables A28 and A29 contain training facilitator and coordinator responses to interview 
questions about the online PD scheduling process. As can be seen in Table A28, participants 
indicated that the schedules were determined by the district (56%). In some cases, the teachers 
were emailed directly to schedule the training (44%), but one participant indicated that teachers 
were not consulted when scheduling training. 


Table A28. Scheduling Process (non-teacher N 

interview) 

Determined by district 5 

Posted on Community of Learning 1 

Emailed teachers 4 

Failed to include teachers 1 


As can be seen in Table A29, over half of the participants indicated that scheduling was 
somewhat difficult (33%) or very difficult (22%). Only one participant indicated that scheduling 
the online training was easy (11%). 


Table A29. Scheduling Ease (non-teacher interview) N 

Easy 1 

Somewhat difficult 3 

Very difficult 2 


Table A30 contains results from interviews with training facilitators and coordinators 
regarding online PD attendance. A common theme was that teachers often arrived late and/or left 
early (44%), often due to classroom coverage issues. It was also commonly mentioned that 
teachers lacked incentive to participate (22%). 


Table A30. Attendance (non-teacher interview) N 

Teachers often late/left early 4 

Lacked incentive 2 

District structure impeded 1 

Technical issues impeded 1 

Attendance good 2 

Attendance poor 2 
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Table A31 contains results from interviews with training facilitators and coordinators 
regarding factors that may have affected online PD attendance. The most common theme was 
that teachers did not have the time to attend or did not want to be away from class (44%). Other 
common themes included a lack of knowledge or expectations about the online PD (22%) and 
uncertainty/reluctance to change from face-to-face to online PD (22%). 


Table A31. Attendance Pressnres (non-teacher N 

interview) 

Lack of knowledge/expectation 2 

Lack time/don't want to be away 4 

Lack common time 1 

Uncertainty/reluctance to change 2 

Skeptical of program’s future 1 

Other 2 


Table A32 contains responses of interviewed nonparticipants of online PD. Nearly half 
(47%) of the nonparticipants indicated that they were not aware that online PD was being 
offered. 

Table A32. Were you aware that web-based professional development was ONP 

being offered? (teacher interview) 

Aware 9 

Not aware 10 

Table A3 3 contains results from teacher interviews regarding the manner in which they 
were contacted about participating in PD. Online participants were more likely to have been 
contacted regarding the PD via email (58%) than in person (23%), whereas nonparticipants were 
more likely to have been contacted in person (53%) than via email (16%). 


Table A33. When and how did you first find out about the professional 
development sessions? Who contacted you about the training? (teacher 

OP 

Online 

ONP 

Online 

F2F 

interview) 

PD 

PD 

PD 

Contacted via email 

18 

5 

3 

Contacted via staff 

7 

0 

10 


Table A34 contains results from teacher interviews regarding when online participants 
and nonparticipants attended PD. Overall, online participants reported that they did so during 
school hours (39%) or their planning period (35%), or outside of school hours (19%). 
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Nonparticipants reported that they attended PD during sehool hours (53%) or that it varied 
(37%), for example attending some PD sessions during the summer and others during sehool 
hours. None of the nonpartieipants reported attending PD during their planning period. 


Table A34. When did you attend training? (teacher interview) 

OP 

ONP 

Total 

During school hours 

12 

10 

22 

During planning period 

11 

0 

11 

Outside school hours 

6 

0 

6 

Summertime or day off 

0 

2 

2 

Varied 

1 

7 

8 


Table A3 5 eontains the results of the teaeher survey regarding when online partieipants 
and nonpartieipants attended PD. Teaehers indicated that they typically participated in faee-to- 
faee PD during teaeher work days (51%) or other days (26%), rather than regular planning time 
(15%) or evenings or weekends (9%). 


Table A35. When did you typically find time to 
participate in PD? (teacher survey) 

Online 

PD 

OP 

F2F PD 
OP ONP 

Total 

Regular planning time 

5 

3 

4 

7 

Teacher work day(s) 

8 

11 

13 

24 

Evenings or weekends 

0 

1 

3 

4 

Other 

6 

5 

7 

12 


Table A3 6 contains results from the teaeher survey regarding the need for substitute 
teaehers to eover elass while teaehers attend PD. Most teaehers reported that substitute teaehers 
were sometimes needed (59%) or always needed (28%) to teaeh their elasses while they attended 
faee-to-faee PD. 


Table A36. How often were substitute teachers 
needed to teach your class while you attended PD? 
(teacher survey) 

Online 

PD 

OP 

OP 

F2F PD 
ONP 

Total 

Never 

9 

2 

4 

6 

Sometimes 

3 

10 

17 

27 

Always 

7 

9 

8 

17 


Table A3 7 eontains results from the teaeher survey regarding the ease of seheduling 
online and faee-to-faee PD. Overall, teaehers slightly disagreed that online PD was easier to 
sehedule than faee-to-faee PD (mean = 3.0). 
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Table A37. Web-based PD was easier to schedule than in-person training. OP 

(teacher survey) 

Strongly Disagree 6 

Moderately Disagree 2 

Slightly Disagree 4 

Slightly Agree 3 

Moderately Agree 4 

Strongly Agree 1 

Mean 3.0 


Table A3 8 contains results from the teacher survey regarding the extent to which online 
participants perceived online PD as more convenient than face-to-face PD in terms of location 
and travel time. Overall, online participants indicated slight agreement that online PD is more 
convenient than face-to-face in terms of location and travel time (mean = 4.1). 


Table A38. Web based PD is more convenient than in-person PD in terms of OP 

location/travel time, (teacher survey) 

Strongly Disagree 3 

Moderately Disagree 0 

Slightly Disagree 4 

Slightly Agree 3 

Moderately Agree 6 

Strongly Agree 4 

Mean 4.1 


Table A3 9 contains results from teacher interviews regarding barriers to attending online 
PD. The most common reasons mentioned were times offered (13%) and being able to get 
substitutes to cover classes (13%). 


Table A39. Were there any other reasons why you might have had trouble OP 

attending the web training session(s) you were invited to? (teacher interview) 

Times offered 4 

Subs/class coverage 4 

Location 2 

No expectation communicated 3 


Table A40 contains results from interviews with training facilitators and coordinators 
regarding the necessity of having an instructor to deliver the PD material. None of the 
participants said that an instructor was needed to deliver the PD. One said that an instructor was 
not needed (11%), and four participants said that the necessity of having an instructor depended 
on the content being delivered (44%). 
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Table A40. Delivery Format (non-teacher N 

interview) 

Instructor needed 0 

Instructor not needed I 

Instructor dependent on content 4 


Expectations/Experiences 

Interviewed teaehers were asked about their thoughts about PD in general, and the results 
are summarized in Table A41. In general, teaehers indieated that PD is important. Online 
partieipants were more likely to indieate that PD is very important (61%) than nonpartieipants 
(32%), whereas nonpartieipants were more likely to eondition the importanee of PD on its 
relevanee (63%) than online partieipants (29%). Two online partieipants said that PD is usually a 
waste of time (6%), and four nonpartieipants said that PD is usually a waste of time (21%). 


Table A41. What are your thoughts related to teacher PD iu geueral? (teacher 
iuterview) 

OP 

ONP 

Total 

Very important 

19 

6 

25 

Important when relevant 

9 

12 

21 

Usually a waste of time 

2 

4 

6 

Other 

0 

1 

1 


Interviewed teaehers provided suggestions for improving PD in their distriet, and the 
results are summarized in Table A42. The most eommon response was better eoordination, 
mentioned by 42% and 47%, respeetively, of online partieipants and nonpartieipants. Teaeher 
input/ehoiee was mentioned by 16% of the online partieipants and 42% of the nonpartieipants. 
More support was mentioned frequently by the online partieipants (32%), and more PD days or 
starting PD before the start of the sehool year were mentioned by 26% of the nonpartieipants. 


Table A42. Overall, what suggestions do you have for improving professional 
development in your district? (teacher interview) 

OP 

ONP 

Total 

Teacher input/choice 

5 

8 

13 

Better coordination 

13 

9 

22 

More support 

10 

1 

11 

No change 

0 

0 

0 

More collaboration 

4 

3 

7 

Prior communication/planning 

2 

1 

3 

More PD days or before school year starts 

2 

5 

7 

Other 

1 

6 

7 
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Table A43 contains the results of teacher interviews regarding the types of PD needed the 
most. Classroom management was mentioned most commonly, by 39% and 37%, respectively, 
of online participants and nonparticipants. Content or program specific PD was also mentioned 
by several teachers, 16% and 32%, respectively, of online participants and nonparticipants. Other 
topics mentioned included instructional strategies (19% of online participants and 16% of 
nonparticipants) and assessment strategies (16% of online participants), and technology (26% of 
nonparticipants). 


Table A43. What sort of PD or PD topics do you think are most needed by you 
and others in your school? (teacher interview) 

OP 

ONP 

Total 

Classroom management 

12 

7 

19 

Instructional strategies 

6 

3 

9 

Assessment strategies 

5 

1 

6 

Engagement/motivation strategies 

4 

1 

5 

Cultural differences 

2 

1 

3 

ELL/ special needs 

3 

2 

5 

T echnology 

2 

5 

7 

Other 

11 

5 

16 

Content or program specific 

5 

6 

11 


Table A44 contains results from teacher interviews regarding teachers’ comfort with 
technology. Overall, most teachers indicated a medium or high level of comfort with technology 
and participating in online PD, with 87% and 95%, respectively, of online participants and 
nonparticipants indicating a medium or high level of comfort. There did not appear to be 
substantial differences between online participants and nonparticipants regarding their comfort 
with technology. 


Table A44. What is your level of comfort with technology and participating in 
training on the web? (teacher interview) 

OP 

ONP 

Total 

High 

9 

7 

16 

Medium 

18 

11 

29 

Low 

3 

1 

4 


Table A45 contains results of the teacher survey regarding teacher confidence in their 
ability to use a computer to learn. Like the interviewed teachers, teachers responding to the 
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survey indicated being moderately confident in their ability to use a computer to learn (mean = 
5.2). 


Table A45. 1 am confident in my ability to nse a compnter to learn, (teacher 
snrvey) 

OP 

ONP 

Total 

Strongly Disagree 

0 

0 

0 

Moderately Disagree 

0 

2 

2 

Slightly Disagree 

0 

1 

1 

Slightly Agree 

7 

3 

10 

Moderately Agree 

4 

9 

13 

Strongly Agree 

10 

16 

26 

Mean 

5.1 

5.2 

5.2 


Level of Satisfaction and Engagement 

Table A46 contains results of the teacher survey regarding teachers’ overall satisfaction 
with the quality of the PD they received. Overall, teachers were moderately satisfied with the 
overall quality of the face-to-face PD (mean = 5.0), and online participants were slightly satisfied 
with the quality of the online PD (mean = 3.8). 


Table A46. Please rate yonr satisfaction with tbe 
overall qnality of PD (teacher snrvey) 

Online 

PD 

OP 

OP 

F2F PD 
ONP 

Total 

Very Dissatisfied 

2 

0 

2 

2 

Moderately Dissatisfied 

3 

0 

1 

1 

Slightly Dissatisfied 

3 

1 

1 

2 

Slightly Satisfied 

2 

2 

2 

4 

Moderately Satisfied 

3 

7 

12 

19 

Very Satisfied 

4 

10 

10 

20 

Not Applicable 

2 

1 

3 

4 

Mean 

3.8 

5.3 

4.8 

5.0 


Table A47 contains responses of interviewed training facilitators and coordinators 
regarding their impressions of online and face-to-face PD. Overall, they had positive impressions 
of online PD. Over half of the training facilitators and coordinators indicated that it has potential 
(67%), that it offers savings (67%) and that it offers flexibility (89%). Other common themes 
included that it allows teachers to participate from a safe and familiar environment (33%), and 
that materials are available on the web for teachers to go back and review later (22%). 
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Table 47. Impressions of online and face-to-face PD N 
(non-teacher interview) 

Web has potential 6 

Web offers savings 6 

Web offers flexibility 8 

Web offers safe/familiar environment 3 

Face-to-face built into district plan 1 

Can go back and review web 2 

Other 7 


Tables A48-A51 contain summaries of teacher interview responses regarding the 
advantages and disadvantages of online and face-to-face PD. Regarding the benefits of online PD 
and aspects of it that teachers like (Table A48), the most common theme was that it is convenient 
(i.e., it saves time, travel, and money) (81%). Teachers also indicted that online PD is engaging 
(55%), it is informative or they liked it (45%), the use of technology is interesting/beneficial 
(42%), it allows teachers to communicate and share with others, both within and outside of the 
school/district (35%), and they can participate in the comfort/perceived safety of their own 
environment (32%). 


Table A48. Advantages / aspects teachers like about web PD (teacher interview) OP 

Informative/liked web PD 14 

Comfort of own environment/non-threatening/non-intimidating 10 

Convenient, saves time, saves money 25 

Engaging 17 

Sharing/communicating with others 1 1 

Technology is interesting/has benefits 13 

Better than face-to-face 2 


Table A49 contains teacher interview responses regarding the disadvantages and aspects 
of online PD that teachers disliked. The most common theme involved problems with technology 
and the resulting time wasted (87%). Other common themes were the lack of planning and/or 
prior notification (68%), the timing of the PD and that teachers were unable to get substitutes to 
cover classes (55%), low participation/lack of personal contact with other teachers (39%), issues 
with trainers (e.g., lack of trainer skill or language issues) (35%). Teachers also indicated that the 
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online PD was not engaging (29%), teaehers needed to be trained on using the teehnology (16%), 
and that they did not like the PD or did not learn anything from it (6%). 


Table A49. Disadvantages / aspects teachers do not like abont web PD (teacher OP 
interview) 

Lack of planning/notification 2 1 

Lack of substitutes/time 17 

Technology issues/wasted time 27 

Lack of teacher skill 5 

Trainer issues 1 1 

Low participation/lack of personal contact 12 

Did not like/did not learn anything 2 

Not engaging 9 


In terms of the advantages of face-to-faee PD and aspects of face-to-face PD that teachers 
liked (Table A50), the most frequently mentioned themes were that it was engaging (95%), the 
personal interaction/collaboration aspect (89%), and that it was informative or they liked it 
(84%). Several teachers also mentioned that the trainers were helpful or better able to gauge a 
face-to-face (63%) and the hands on/modeling aspect of face-to-face PD (47%). 


Table A50. Advantages / aspects teachers like about face-to-face PD (teacher ONP 

interview) 

Informative/liked face-to-face PD 16 

Engaging 1 8 

Hands-on/modeling 9 

Personal interaction/collaboration 17 

Trainers helpful/better able to gauge audience 12 


In terms of disadvantages and aspects of the face-to-face PD that teachers did not like 
(Table A51), the most common theme was that the PD was not well planned, including that the 
timing was not ideal and that there were issues with adequate prior notification of the training 
(74%). 


Table A51. Disadvantages / aspects teachers do not like about face-to-face PD ONP 
(teacher interview) 

Not engaging 1 

Lack of planning/timing/notification 14 

Did not like/did not learn anything 2 


Interviewed teachers were asked for suggestions to improve online PD, and a summary of 
their responses are presented in Table A52. Teachers’ top suggestions were prior 



79 


planning/preparation (23%) and advance training on the teehnology (23%), followed by setting 
clear expectations (16%). 


Table A52. What suggestions do you have for improving the web training? OP 

(teacher interview) 

Advance training 7 

Offer in summer or earlier in year 1 

Set clear expectations 5 

Prior planning/preparation 7 

Blended solution 1 

Incorporate teacher uploads 1 

Increase accountability 2 

Other 5 


Table A53 contains suggestions from interviews with training faeilitators and 
eoordinators for how to improve online PD. The most eommon theme was related to training 
teaehers on the web environment (89%), followed by providing clear expeetations (78%), and 
offering a blended solution, where PD starts faee-to-faee, then moves online (67%). Other 
suggestions ineluded that aeeountability was needed to ensure teaeher partieipation (44%), 
inereased flexibility (22%), scheduling the training in advanee (22%), and redesigning the 
training for the web rather than merely presenting the same material as would be presented faee- 
to-faee over the web (22%). It was also suggested that online PD should be offered on a broader 
seale, regionally or nationally (22%). 


Table A53. Web Improvements (non-teacher N 

interview) 

Accountability needed 4 

Increase flexibility 2 

Schedule in advance 2 

Provide clear expectations 7 

Train on environment/involve IT 8 

Offer blended solution 6 

Redesign training for web 2 

Offer regionally/nationally 2 

Other 8 


Table A54 contains results from the teacher survey regarding the pereeived effeetiveness 
of online and faee-to-faee PD. Overall, online partieipants indieated slight to moderate 
agreement that online PD is less effective than face-to-face PD (mean = 4.5). 



80 


Table A54. Web-based PD is less effective than in-person training, (teacher OP 

snrvey) 

Strongly Disagree 0 

Moderately Disagree 3 

Slightly Disagree 3 

Slightly Agree 2 

Moderately Agree 4 

Strongly Agree 7 

Mean 4.5 


Table A55 contains teacher interview responses to whether they preferred or were open 
to online, face-to-face, or blended PD. Overall, teachers indicated that they preferred face-to-face 
PD, with 61% and 53%, respectively, of online participants and nonparticipants indicating that 
they would rather participate in face-to-face PD. Two teachers in each group indicated that they 
preferred online PD. One online participant (3%) and seven nonparticipants (37%) indicated that 
they would prefer or be open to a blended solution, in which the first session(s) of the PD were 
face-to-face, and then subsequent sessions would be offered online. 

Table ASS. Prefer/open to web, face-to-face, or blended PD (teacher interview) OP ONP 

Web 2 2 

Face-to-face 19 10 

Blended (start with face-to-face, then move to web) 1 7 

Table A56 contains results from the teacher survey regarding whether online participants 
would have preferred to have face-to-face PD. Overall, teachers indicated moderate agreement 
that they would rather have had face-to-face PD (mean = 5.1). 


Table AS6. All things considered, I wonld rather have had in-person training. OP 

(teacher snrvey) 

Strongly Disagree 0 

Moderately Disagree 1 

Slightly Disagree 1 

Slightly Agree 3 

Moderately Agree 4 

Strongly Agree 10 

Mean 5.1 


Online participants interviewed were asked if the online PD format had an impact on 
their ability to learn the material covered in PD, and the results are presented in Table A57. Over 
half of the online participants indicated that the online format had no impact on their ability to 
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learn the material eovered (52%). Of those who said it did impaet their ability to learn, a 
common explanation was that technology issues interfered (29%). 


Table A57. Do you think the fact that the training was web-based had an OP 

impact on your ability to learn the material being covered in the training? 

(teacher interview) 

No impact on ability to learn 16 

Tech/audio distractions 9 

No materials 1 

Language barrier 5 

Limited time 3 


Table A58 contains results of the teacher survey regarding the extent to which 
participating in online PD enhanced teachers’ desire to attend future online PD. Overall, teachers 
indicated that they were not interested in attending future online PD (mean = 2.7). 


Table ASS. Participation in web-based PD enhanced my desire to attend future OP 

training sessions, (teacher survey) 

Strongly Disagree 9 

Moderately Disagree 1 

Slightly Disagree 2 

Slightly Agree 2 

Moderately Agree 3 

Strongly Agree 2 

Mean 2.7 


Table A59 contains results from the teacher survey regarding the extent to which online 
participants would recommend online PD to a colleague. Overall, teachers indicated that they 
would not recommend online PD to a colleague (mean = 3.2). 


Table A59. 1 would recommend the web-based PD to a colleague, (teacher OP 

survey) 

Strongly Disagree 5 

Moderately Disagree 2 

Slightly Disagree 2 

Slightly Agree 7 

Moderately Agree 4 

Strongly Agree 0 

Mean 3.2 


Training facilitators and coordinators interviewed were asked about teacher engagement 
during online PD, and the results are presented in Table A60. Four participants indicated that 
teachers seemed engaged in the online PD (44%). Two indicated that sharing and collaboration 
between teachers resulted in more engagement (22%). Three participants indicated that the small 
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numbers of teachers participating in online PD groups hindered teacher engagement in the PD 
(33%), and one indicated that technology problems hindered teacher engagement (11%). 


Table A60. Engagement (non-teacher interview) N 

Seemed engaged 4 

Difficult to judge 1 

Technology problems hindered 1 

Small numbers hindered 3 

Sharing/collaboration good 2 


Table A61 contains teacher interview responses regarding their level of engagement in 
the PD. Overall, most teachers indicated that they found the PD to be engaging or very engaging 
(55% and 95%, respectively, of online participants and nonparticipants). 


Table A61. Did you find training to be engaging? (teacher interview) 

OP 

ONP 

Total 

Very engaging 

4 

5 

9 

Engaging 

13 

13 

26 

Not as engaging as expected 

4 

0 

4 

Not engaging 

7 

1 

8 

Not as engaging as face-to-face 

2 

0 

2 

Already had this training 

1 

0 

1 


Surveyed teachers were also asked about their level of engagement in the PD, and the 
results are presented in Table A62. Overall, teachers indicated moderate agreement that the PD 
actively engaged them in reflecting on their teaching (mean = 5.0). 


Table A62. Tbe PD actively engaged me in reflecting on my teaching, (teacher 
survey) 

OP 

ONP 

Total 

Strongly Disagree 

1 

0 

1 

Moderately Disagree 

1 

3 

4 

Slightly Disagree 

0 

0 

0 

Slightly Agree 

2 

4 

6 

Moderately Agree 

6 

11 

17 

Strongly Agree 

11 

13 

24 

Mean 

5.1 

5.0 

5.0 


Table A63 contains results from interviews with training facilitators and coordinators 
regarding their impressions of the technology in schools participating in online PD. Most 
participants believed that the technology available to teachers for the online PD was adequate 
(89%). However, there were some issues mentioned with respect to technology problems, 
including connectivity and login issues (56%), room configuration/phone line issues (33%), 
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headset/audio issues (33%), and that these issues hindered the start of the PD (22%). Four 
partieipants indieated that teaehers laeked skill or comfort with the technology at first (44%), but 
it was also mentioned that skill and comfort improved over time (33%). 


Table A63. Technology Impressions (non-teacher N 

interview) 

Technology available was adequate 8 

Room configuration/phone line issue 3 

Headset/hearing issue 3 

Connectivity/login issue 5 

Technology slowed start 2 

Skills/comfort lacking 4 

Skills/comfort improved over time 3 

Other 2 


Table A64 contains interviewed teachers responses regarding whether technology 
problems deterred them from attending online PD. Overall, 10% of online participants and 21% 
of nonparticipants indicated that technology problems prevented them from attending the online 
training. 


Table A64. Did technology problems ever deter you from attending web 
training? (teacher interview) 

OP 

ONP 

Total 

Yes 

3 

4 

7 

Somewhat reluctant 

2 

0 

2 

No 

13 

3 

16 


Online participants were asked if they experienced any difficulties with technology 
during training, and the results are presented in Table A65. The most common technological 
problem mentioned by online participants was problems with the audio (52%). Configuration 
and computer equipment problems were also mentioned by 26% of the teachers, and connectivity 
problems were mentioned by 23% of the teachers. 

Table A65. During training, did you experience any difficulties with the OP 

technology? (teacher interview) 


Configuration 5 

Connectivity 7 

Equipment - PC/tools 3 

Equipment - Audio 1 6 

Teacher lack of skill/comfort 2 

Trainer lack of skill/comfort 1 

Other 4 
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Online participants were also asked if the technology problems had an impact on their 
ability to learn, their opinion of online PD, and their likelihood of attending more online PD. 
Results are presented in Table A66. Overall, 29% of the online participants indicated that their 
ability to learn was impacted, and 23% indicated that the technology problems impacted their 
opinion of online PD. A slightly smaller number of teachers (16%) indicated that the technology 
problems impacted their willingness to participate in additional online PD. 


Table A66. What impact did your technology problems have on: Your ability 

OP 

to learn? Your opinion of web training? Your likelihood of attending more 

Impact 

No 

web training? (teacher interview) 


Impact 

Ability to learn 

9 

6 

Opinion of web training 

7 

8 

Likelihood of attending more web training 

5 

8 


Learning, Transfer, and Outcomes 


Table A67 contains responses from teacher interviews regarding the impact of the PD. In 
general, teachers indicated a positive influence of the PD, with 55% and 89%, respectively, of 
the online participants and nonparticipants indicating being influenced positively, including new 
ways of teaching and new ways of learning. 


Table A67. How do you think the PD you participated in has influenced you? 
(teacher interview) 

OP 

ONP 

Total 

Positively 

6 

5 

11 

New ways of teaching 

8 12 

20 

New ways of learning 

3 

0 

3 

No influence 

2 

0 

2 


Table A68 contains teacher interview responses regarding the impact of PD on teacher 
knowledge in various areas of their profession. Online participants were more likely to indicate 
that the PD did not have an impact than nonparticipants. Online participants were most likely to 
indicate that the PD had an impact on addressing student misconceptions (55%) and teacher 
collaboration (55%), followed by use of assessment data (52%) and participation in PLCs (52%). 
Nonparticipants were most likely to indicate that the PD had an impact on use of assessment data 
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(74%) and addressing student misconceptions (74%), followed by teacher collaboration (68%) 
and other classroom activities (68%). 


Table A68. Did the PD affect how much 


OP 



ONP 


knowledge you have about... (teacher 

Impact 

No 

Can’t 

Impact 

No 

Can’t 

interview) 


Impact 

Judge 


Impact 

Judge 

The subject area you teach? 

14 

12 

1 

5 

9 

0 

Classroom management skills? 

15 

15 

0 

12 

6 

0 

Use of assessment data? 

16 

10 

2 

14 

6 

0 

Addressing student misconceptions? 

17 

10 

1 

14 

5 

0 

Any other classroom activities? 

12 

14 

0 

13 

3 

0 

T eacher collaboration? 

17 

13 

0 

13 

6 

0 

Professional learning communities? 

16 

10 

1 

11 

5 

0 

Any other areas? 

5 

18 

0 

4 

7 

0 


Table A69 contains the results of teachers interviewed regarding the extent to which they 
believed that implementation of the program for which they received PD would result in 


improved student learning. Overall, 58% and 74%, respectively, of online participants and 


nonparticipants indicated that they believed it would improve student learning. 



Table A69. Overall, do von think vour implementation of 
will, result in improved student learning? (teacher interview) 

has, or 

OP 

ONP 

Total 

Improved learning 


18 

14 

32 

Improved understanding 


2 

0 

2 

Improved behavior 


1 

0 

1 

Improved performance 


5 

5 

10 

No change 


3 

0 

3 

Can’t judge 


5 

3 

8 


Table A70 contains results from the teacher survey regarding the extent to which teachers 
believe that the PD will result in greater student learning. Overall, teachers indicated moderate 
agreement that the PD will result in greater student learning (mean = 5.1). 


Table A70. 1 believe that the PD I received will eventually result in greater 
student learning, (teacher survey) 

OP 

ONP 

Total 

Strongly Disagree 

0 

1 

1 

Moderately Disagree 

0 

1 

1 

Slightly Disagree 

0 

2 

2 

Slightly Agree 

3 

5 

8 

Moderately Agree 

7 

10 

17 

Strongly Agree 

11 

12 

23 

Mean 

5.4 

4.9 

5.1 


Online participants interviewed were asked whether they believed that their participation 
in online PD would impact their ability to implement the program for which they received PD, 
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and the results are presented in Table A71. Of the online partieipants, 35% indieated that their 
partieipation in the online PD had a positive impact. Eight teachers (26%) indicated that it had no 
impact, and one teacher indicated that it had a negative impact, due to multiple issues with 
scheduling, class coverage, and technology problems. 


Table A71. Do you think your participation in web-based PD for OP 

impacted your ability to implement ?(teacher interview) 

Positive impact 1 1 

Slight impact 6 

No impact 8 

Can’t judge 1 

Negative impact 1 


Table A72 contains results from the teacher survey regarding whether teachers reported 
implementing the skills they learned during PD. Overall, teachers agreed moderately that they 
had begun implementing the skills they learned during PD (mean = 5.1). 


Table A72. 1 have begun implementing the skills I learned during PD. (teacher 
survey) 

OP 

ONP 

Total 

Strongly Disagree 

1 

0 

1 

Moderately Disagree 

0 

2 

2 

Slightly Disagree 

0 

0 

0 

Slightly Agree 

3 

5 

8 

Moderately Agree 

7 

9 

16 

Strongly Agree 

10 

15 

25 

Mean 

5.1 

5.1 

5.1 
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Appendix B: Coding System for Interviews 


Interviews with study subjeets, ineluding teaehers, training facilitators, training 
coordinators, and programs leaders, were transcribed. Using software for qualitative data 
analysis (Muhr & Friese, 2004), responses were then categorized by applying codes to the 
transcribed information. Appendix B contains the list of codes for the teacher and non-teacher 
interviews. 

Teacher Interview Codes 
Background 


Si.l subject 


[subject(s) taught] 

Si.1.1 

math 


Si. 1.2 

English 


Si.l years 


[total years teaching] 

Si.2.1 

1-2 yrs 


Si.2.2 

3-5 yrs 


Si.2.3 

6-9 yrs 


Si.2.4 

10-19 yrs 


Si.2.5 

20+ yrs 



51.3 technology comfort 

51. 3.1 prior level comfort high 

51. 3.2 prior level comfort med 

51. 3.3 prior level comfort low 

51.4 program taught 

51.4.1 QCA/Math 1 

51.4.2 Math Navigator 

51.4.3 Ramp-Up Literacy 

51.5 program PD 

51. 5.1 yes 

51.5.2 no 

51.6 other PD 

51.6.1 QCA/Math 1 

51. 6.2 Math Navigator 

51. 6. 3 Ramp-Up Literacy 

51. 6.4 Study groups 

51. 6. 5 other 

51.7 PD reccived 

51.7.1 f2f 

51. 7.2 web 

51.7.3 both 


[individual’s general level of comfort with technology] 


[program taught] 


[PD received for program] 


[PD for other programs] 


[type of PD received] 
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0. logistics support [training logistics and support] 

0. 1 . contact_email 

0.2. contact_staff 

0. 3. contact_reminder 

la.l contacted_district staff 

la.2 contacted_AC staff 

la. 3 contacted_both 

l b. l had leadership support 

lb. 2 lacked leadership support 

lb. 3 n/a or don’t recall 

1 c . 1 training required 

lc. 2 training optional 

lc. 3 not sure/don’t recall 

1. C.4 recommended/encouraged 

l d. l colleague support 

ld.2 no colleague support 

ld.3 n/a or don’t recall 

l.d.4 split (some colleague support, others not) 

1. notiflcation 

1.1. enough prior notification 

1 .2. not enough prior notification 

1.3. don’t recall 

1.4. sometimes had enough notice 

2. scheduling conflicts [barriers to scheduling training] 

2.1. no subs/class coverage 

2.2. poor notification 

2.3. times offered 

2.4. lack consistent schedules 

3. technology deterrent [technology deter from attending] 

3.1. yes 

3.2. somewhat reluctant 

3.3. no 

4. other deterrent [barriers to attending training] 

4. 1 . times offered 

4.2. don't want to be out of class 

4.3. subs/class coverage 

4.4. location 

4.5. no notification 

4.6. late planning 

4.7. no time 

4.8. predisposed feelings (e.g., prefer f2f) 

4.9. no expectation communicated 

5. technology difficulties 

5.1. configuration (e.g., PCs not setup properly in advance) 

5.2. connectivity 

5.3. equip_PC/tools 

5.4. equip_audio (e.g., no headphones, no phone line near PC 

5.5. teacher lack skill/comfort 

5.6. trainer lack skill/comfort 

5.7. other (e.g., password issues, etc.) 


[technology barriers during training] 


— poor room configuration) 
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6a. 1 difficulties resolved 

6a,2 difficulties not resolved 

6a,3 took too long to resolve 

6b. 1 impacted ability to learn 

6b.2 did not impact ability to learn 

6b. 3 impacted opinion 

6b.4 did not impact opinion 

6b. 5 impacted retention 

6b. 6 did not impact retention 

6. when attended [when attended training] 

6. 1 . during school hours 

6.2. during planning period 

6.3. outside school hours 

6.4. summertime or day off 

6.5. varied 

7. where attended 

7. 1 . classroom/office 

7.2. offsite facility 

7.3. computer lab 

7.4. had to find space 

7.5. assigned space 

8. number sessions 

8 . 1 . 0-1 

8 . 2 . 1-2 

8.3. 3-4 

8.4. 5+ 

8a. 1 attended one-to-one tutoring 

8b. 1 improve attendance (e.g., anything might have improved attendance or ability to attend) 


8b.l.l 

more offerings 

8b. 1.2 

more notice 

8b. 1.3 

subs/class coverage 

8h.l.4 

other (e.g., organization, expectations) 


9. web opinions [overall opinions of web training] 

9. 1 . informative 

9.2. better than f2f 

9.3. prefer f2f^hands on 

9.4. comfortable/non-threat environment 

9.5. didn’t take time away from class 

9.6. too much time wasted (e.g., tech difficulties, getting up to speed/logged on) 

9.7. lacked materials (e.g., not receiving materials in advance made difficult to participate) 

9.8. was not able to learn 

9.9. need more flexible times 

9. 10. needed more participants 

9.11. trainers were having difficulties 

9.12. liked online training 

9. 13. did not like online training 

9.14. other 

10. web engagement [was web training engaging] 

10.1. very engaging 

10.2. somewhat engaging 

10.3. not as engaging as expected 

10.4. not engaging 

10.5. not as engaging as f2f 

10.6. already had this training f2f 


[where individual participated from] 


[approx number of sessions attended] 
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10.7. can’t judge (e.g., too many problems, too few participants, too few sessions) 

11. web liked [things liked about web training] 

11.1. not missing class 

11.2. not traveling 

1 1 .3 . sharing examples/ideas 

1 1 .4. access to others outside (e.g., hearing ideas, opinions, etc. from people outside my school/district) 

1 1 .5. comfort of own surroundings 

11.6. less intimidating 

11.7. technology interesting/adventurous 

11.8. less time required 

11.9. flexibility 

11.10. other 

12. web didn’t like [things didn’t like about web training] 

12.1. wasted time (e.g., time wasted on technology and login) 

12.2. lack of options (e.g., would have liked more time options) 

12.3. lack of personal/human contact 

12.4. lack of attendees 

12.5. missing advance materials 

12.6. system not ready (e.g., computer systems/configurations not set up properly in advance) 

12.7. mixed messages (e.g., poor communication — ^who does what, where direct questions) 

12.8. language issues with trainers 

12.9. sessions not long enough 

12.10. other 

13. web advantages [advantages of web training] 

13.1. own environment 

13.2. more flexible 

13.3. more engaging 

13.4. safer (e.g., less intimidating, free to ask questions) 

13.5. time/travel savings 

13.6. cost savings 

13.7. collaboration 

13.8. great potential 

13.9. afford train new TRs (e.g., web makes it possible to provide full set of training to new teachers) 

13.10. record review (e.g., web affords ability to record sessions, ability to review sessions later) 

13.11. try things return (e.g., web allows TRs to be trained, try it out, and return for future assistance) 

13.12. individualized training 

14. web prefer f2f delivery [would have preferred training be delivered f2f] 

14.1. prefer f2f 

14.2. more intimidating 

14.3. tech distractions 

14.4. less flexible 

15. web improvements [suggestions for improving web training] 

15.1. blended solution (e.g., begin with f2f training to build community, then move to web for followup) 

15.2. copilot manage environment (e.g., trainer could use a copilot to help manage the tech environment) 

15.3. ensure TR credit 

15.4. incorporate TR uploads (e.g., allow teachers to upload videos, assignments, questions in — maybe in advance) 

15.5. increase accountability (e.g., make return with assignments, attendance reports for administration) 

15.6. increase collaboration 

15.7. more resource links 

15.8. offer regionally 

15.9. schedule well advance 

15.10. take program to scale (e.g., avoid implementing with “model classrooms”) 

15.11. advance training (e.g., have a walkthrough of techno logy/environment in advance of first session) 

15.12. offer in summer or earlier in year 

15.13. set clear expectation 

15.14. other 

15.15. prior planning/preparation 
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16. web learning barriers [barriers that made learning difflcult or impossible] 

16.1. no impact on ability to learn 

16.2. tech/audio distractions 

16.3. low attendance (e.g., too few participants to be of value) 

16.4. too few sessions (e.g., only held/attended one session) 

16.5. no materials (e.g., materials were not received in advance of the training) 

16.6. language barrier (e.g., tutors non-native English speakers) 

16.7. limited time 

17. web influence [bas web training influenced participant] 

17.1. positively 

1 7.2. new ways of teaching 

17.3. new way of learning (e.g., excited for technology, online training) 

17.4. no influence 

17.5. varied widely 

18a. 1 knowledge impact 
18a.2 no knowledge impact 

18a.3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

18h.l management impact 
18h.2 no management impact 

18b. 3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

18c.l data impact 
18c. 2 no data impact 

18c. 3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

18d.l misconcept impact 
18d.2 no misconcept impact 

18d.3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

18e.l other activities impact 
18e.2 no other activities impact 

18e.3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

18f. 1 collaboration impact 
18f.2 no collaboration impact 

18f 3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

18g.l PLC impact 
18g.2 no PLC impact 

18g.3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

18h.l other areas impact 
18h.2 no other areas impact 

18h.3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

18. web expert access [extent bad access to program experts outside of training] 

18.1. good access 

18.2. no access 

18.3. access was better last year 

18.4. limited access 


19 . 


web implementation barriers [anything keeping them from implementing wbat they learned] 

19.1. lacking time 

19.2. lacking knowledge (e.g., didn’t learn enough during training to be useful) 

19.3. lacking materials (e.g., district did not purchase materials for all) 

19.4. no longer teaching program (e.g., shift in teaching responsibilities) 

19.5. lacking support 

19.6. student behavior issues 
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20. importance of implementation [importance of fully implementing program] 

20.1. very important 

20.2. skeptical (e.g., just another program) 

20.3. tom (e.g., confused by lack of support, state test issues) 

20.4. no expectation (e.g., lacked prior knowledge to judge) 

20.5. initial infringement (e.g., initially felt was an infringement to tell me how/what to teach) 

20.6. not very important 

2 1 a. 1 very supportive 

2 1 a. 1 . 1 provided time to attend 

2 1 a. 1 .2 provided time to implement 

2 1 a. 1 .3 provided resources 

2 1 a.2 skeptical (e.g., just another program) 

21a.3 not involved enough 

21a.4 poor communication 

21a.5 lacked resources/support 

21b. 1 good fit 

2 lb. 2 not good fit 

21b. 3 no expectation (e.g., lacked prior knowledge to judge) 

21b.4 need lower program (e.g., need program for lower performing students) 

21c.l supportive 

21c. 2 not supportive 

21c. 3 n/a or don’t recall 

21c.4 skeptical 

2f. student outcome [impact program had/will have on students] 

21.1. improved learning 

21.2. improved understanding 

21.3. improved behavior 

21.4. improved performance 

21.5. no change 

21.6. can’t judge 

22. web impact ability to implement [did participation in web training impact ability to implement] 

22.1. positive impact 

22.2. slight impact 

22.3. no impact 

22.4. can’t judge 

22.5. negative impact 

23. general PD perception [general perception of PD] 

23.1. very important 

23.2. important when relevant 

23.3. usually a waste of time 

24. improve district PD [thoughts on improving PD within district] 

24. 1 . teacher input/choice (e.g., want more flexibility in choosing what PD they need) 

24.2. better coordination (e.g., more thought and coordination into what have received/what need) 

24.3. more support 

24.4. no change 

24.5. more collaboration 

24.6. prior communication/planning 

24.7. more PD days/before schoolyear 

24.8. other 


25. PD topics 

25.1. classroom management 

25.2. instructional strategies 


[PD topics needed by teacher and colleagues] 
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25.3. assessment strategies 

25.4. engagement/motivation strategies 

25.5. cultural differences 

25.6. ELL/special needs 

25.7. technology 

25.8. content or program specific 

25.9. other 


26. web awareness topics 

26.1. web has potential 

26.2. excited about web opportunity 

26.3. TRs need to learn how to participate 

26.4. experience could be tweaked 

26.5. develop p Ian that works 

26.6. flexibility in scheduling 

26.7. make sure things working 

26.8. R&R is working 

26.9. other 


[other things regarding web training we need to be aware of] 


Online Web Nonparticipants 


np.l np logistics [non-web participant training logistics and support] 

npl.l aware 

npl.2 not aware 

npla.l contact_email 

npla.2 contact_staff 

npla. 3 contact_reminder 

nplb. l contacted_district staff 

nplb.2 contacted_AC staff 

nplb. 3 contacted_both 

nplc. l when contacted 

np 1 d. 1 enough prior notification 

npld. 2 not enough prior notification 

npld.3 don’t recall 

np.l scheduling conflicts [barriers to scheduling training] 

np2.1 no subs/class coverage 

np2.2 poor notification 

np2.3 times offered 

np2.4 lack consistent schedules 

np.3 technology deterrent [technology deter from attending] 

np3.1 yes 

np3.2 somewhat reluctant 

np3.3 no 

np.4 leadership support [was leadership supportive of attending web training] 

np4.1 good leadership support 

np4.2 lacked leadership support 

np4.3 n/a or don’t recall 

np4.4 training required 

np4.5 training optional 

np4.6 not sure/don’t recall 


np.5 material repeat 

np5.1 yes 


[was material being covered in web training info already knew] 
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np5.2 no 

np5.3 don’t know 

np5.4 f2f training was adequate 

np.6 reasons not interested [why was TR not interested in attending web training] 


np6.1 

didn’t need it 

np6.2 

topic not of interest 

np6.3 

didn’t know topic 

np6.4 

it was optional 

np6.5 

didn’t know expectation 

np6.6 

other 


np.7 increased desire [what would have made TR more likely to attend web training] 


np7.1 

offered f2f 

np7.2 

more leadership support 

np7.3 

more colleague support 

np7.4 

CE credits 

np7.5 

other 


Face-to-face 


Of.8 Of logistics 

f2f8.1 contact_email 

f2f8.2 contact_staff 

f2f8.3 contact_reminder 

f2f8a. 1 contacted_district staff 

f2f8a,2 contacted_AC staff 

f2f8a.3 contacted_both 

f2f8b.l when contacted 

f2f8c. 1 enough prior notification 

f2f8c.2 not enough prior notification 
f2f8c.3 don’t recall 

f2f8c.4 sometimes had notice 

Of.9 scheduling conflicts 

f2f9.1 no subs/class coverage 

f2f9.2 poor notification 

f2f9.3 times offered 

f2f9.4 lack consistent schedules 

f2f.l0 leadership support 

f2fl0.1 good leadership support 

f2fl0.2 lacked leadership support 

f2fl0.3 n/a or don’t recall 

f2fl0a.l training required 

f2fl0a.2 training optional 

f2fl0a.3 not sure/don’t recall 

f2fl0a,4 other 

f2f.ll when attended 

f2fl 1 . 1 during school hours 

f2fl 1 .2 during planning period 

f2fl 1 .3 outside school hours 

f2fl 1 .4 summertime or day off 

f2fll.5 varied 


[non-web participant training logistics and support] 


[barriers to scheduling training] 


[was leadership supportive of attending web training] 


[when attended f2f training] 
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t2f.ll where attended [where individual participated from] 


f2fl2.1 

my school 

f2fl2.2 

another school 

f2fl2.3 

offsite facility 

f2fl2.4 

traveled 1-9 minutes 

f2fl2.5 

traveled 10-19 minutes 

f2fl2.6 

traveled 20+ minutes 

f2fl2.7 

varied 


f2f.l3 f2f_number sessions 

[approx number of f2f sessions attended] 

f2fl3.1 

0-1 


f2fl3.2 

1-2 


f2fl3.3 

3-4 


f2fl3.4 

5+ 


f2f.l4 f2f_opinions 

[overall opinions of f2f training] 

f2fl4.1 

informative 


f2fl4.2 

too long (e.g., session was longer than necessary) 


f2fl4.3 

timing was off (e.g., timing of training did not fit with teaching of curriculum) 

f2fl4.4 

was not able to learn 


f2fl4.5 

prefer web 


f2fl4.6 

liked it 


f2fl4.7 

did not like it 



f2f.l5 f2f_engagement [was f2f training engaging] 

f2fl 5 . 1 very engaging 

f2fl5.2 somewhat engaging 
f2fl5.3 not as engaging as expected 

f2fl5.4 not engaging 

f2fl5.5 can’t judge (e.g., too many problems, too few participants, too few sessions) 

f2f.l6 f2f_liked [things liked about f2f training] 

f2fl6.1 modeling 
f2fl6.2 Interactions/sharing 
f2fl6.3 trainers good/helpfiil 

f2f.l7 f2f_advantages [advantages of f2f training] 


f2fl7.1 

personal setting 

f2fl7.2 

real time response 

f2fl7.3 

easier to communicate 

f2fl7.4 

hands-on/ model ing 

f2fl7.5 

trainers better able to gauge 

f2fl7.6 

collaboration 

f2fl7.7 

other 


f2f.l8 f2f_didn’t like [things didn’t like about f2f training] 

f2fl8.1 wasted time (e.g., time wasted stretching material or on tech issues) 

f2fl8.2 overwhelming content 

f2fl8.3 late start/felt rushed 

f2fl8.4 technology glitches 

f2fl8.5 other 


f2f.l9 f2f_needed f2f delivery 

f2fl9.1 yes 
f2fl9.2 only portion 
f2fl9.3 no 


f2f.20 f2f_influence [has web training influenced participant] 

f2f20.1 positively 

f2f20.2 new ways of teaching 

f2f20.3 new way of learning (e.g., excited for technology, online training) 

f2f20.4 no influence 
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f2f20.5 varied widely 

f2f20a. 1 knowledge impact 
f2120a.2 no knowledge impact 

f2120a.3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

fZfZOb.l management impact 
f2120b.2 no management impact 

f2f20b.3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

fZfZOc.l data impact 
f2120c.2 no data impact 

f2f20c.3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

f2120d. 1 misconcept impact 
f2f20d.2 no misconcept impact 

f2f20d.3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

f2120e.l other activities impact 
f2f20e.2 no other activities impact 

f2f20e.3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

f2f20f. 1 collaboration impact 
f2f20f.2 no collaboration impact 

fZfZOf 3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

fZfZOg.l PLC impact 
f2f20g.2 no PLC impact 

f2f20g.3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

fZfZOh. 1 other areas impact 
f2f20h.2 no other areas impact 

f2f20h.3 can’t judge (e.g., too many problems, too few participants, too few sessions) 

Of.21 Of expert access [extent had access to program experts outside of f2f training] 

f2f21.1 good access 

f2f21.2 no access 

f2f2 1 .3 access was better last year 

f2f2 1 .4 limited access 

f2f.22 f2f_implementation barriers [anything keeping them from implementing what they learned] 

12122. 1 lacking time 

f2f22.2 lacking knowledge (e.g., didn’t learn enough during training to be useful) 

12122.3 lacking materials (e.g., district did not purchase materials for all) 

12122.3 no longer teaching program (e.g., shift in teaching responsibilities) 

12122.4 student behavior issues 

f2f.23 f2f_importance of implementation [importance of fully implementing program] 

12123 . 1 very important 

12123.2 skeptical (e.g., just another program) 

12123.3 tom (e.g., conliised by lack of support, state test issues) 
f2123.4 no expectation (e.g., lacked prior knowledge to judge) 

12123.5 initial infringement (e.g., initially felt was an infringement to tell me how/what to teach) 

f2123a.l very supportive 

12f23a. 1 . 1 provided time to attend 

12f23a. 1 .2 provided time to implement 

f2f23a. 1 .3 provided resources 

12f23a.2 skeptical (e.g., just another program) 
f2f23a.3 not involved enough 
12f23a.4 poor communication 
12123a.5 more supportive last year 
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f2f23b.l 

good fit 


f2f23b,2 

not good fit 


f2f23b.3 

no expectation (e.g., lacked prior knowledge to judge) 


f2f23b.4 

need lower program (e.g., need program for lower performing students) 

f2f23b.5 

skeptical 


f2f23c.l 

very supportive 


f2f23c,2 

not supportive 


f2f23c.3 

n/a or don’t recall 


f2f23c.4 

skeptical 


f2f23c,5 

split (some colleague support, others not) 


f2f.24 flf student outcome 

[impact program had/wiii have on students] 

f2f24.1 

improved learning 


f2f24.2 

improved understanding 


f2f24.3 

improved behavior 


f2f24.4 

improved performance 


f2f24.5 

no change 


f2f24.6 

can’t judge 


f2f.25 f2f_general PD perception 

[generai perception of PD] 

f2f25.1 

very important 


f2f25.2 

important when relevant 


f2f25.3 

can be waste of time 


f2f25.4 

other 


f2f.26 f2f_improve district PD 

[thoughts on improving PD within district] 

f2f26.1 

teacher input/choice (e.g., want more flexibility in choosing what PD they need) 

f2f26.2 

better coordination (e.g., more thought and coordination into what have received/what need) 

f2f26.3 

more support 


f2f26.4 

no change 


f2f26.5 

other 


f2f26.6 

more collaboration 


f2f26.7 

prior communication/planning 


f2f26.8 

more PD days/before schoolyear 


f2f.27 f2f PD 

topics 

[PD topics needed hy teacher and coiieagues] 

f2f27.1 

classroom management 


f2f27.2 

instructional strategies 


f2f27.3 

assessment strategies 


f2f27.4 

engagement/motivation strategies 


f2f27.5 

cultural differences 


f2f27.6 

ELL/special needs 


f2f27.7 

technology 


f2f27.8 

other 


f2f27.9 

content or program specific 



T1 programs change frequently 


Study Group Teacher Interview Codes 
Background 


1 study group [participate in which study group] 

1 . 1 Team roles and Responsibilities 

1 .2 Behavioral readiness - overview 

1 .3 ENGAGE (formerly known as Student Readiness Inventory) 

1 .4 ENGAGE Teacher Edition (formerly known as Behavioral Monitoring Scales) 
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1.5 

EXPLORE Reports 



1.6 

Career Exploration and Planning 


2 

subject 

& grade levels 

[subject(s) taught] 


2.1 

English 



2.2 

history 



2.3 

math 



2.4 

science 


3 

years 


[total years teaching] 


3.1 

< 5 yrs 



3.2 

> 10+ yrs 


4 

technology comfort 

[individual’s general level of comfort with technology] 


4.1 

prior level comfort high 



4.2 

prior level comfort med 



4.3 

prior level comfort low 


5 

programtaught 

[program taught or involved] 


5.1 

QCA/Math 1 



5.2 

MNAV 



5.3 

RUL 



5.4 

Behavior Readiness Study Groups 



5.5 

Other 



Interview Focus 


1. study group [how did you hear about study groups] 

la.l facilitator (school employee) 

la.2 principal 

la. 3 AC or ACT employee 

1 b . 1 very supportive 

lb. 2 yes 

2. Decide to attend 

2. 1 . know what the students are doing 

2.2. PD benefit for teachers 

2.3. get on the same page as other teachers 

2.4. required 

3. Organizer responsible 

3.1. multiple people 

3.2. America’s Choice program manager 

3.3. facilitator 

4. Problems attending sessions 

4. 1 . No problems 

5. Time attended sessions [when you did you attend sessions] 

5.1. during designated team time 

6. How often did you meet to talk about topic area 

6.1. Yes 

7. Visited COL website? 

7.1. Yes 

7.2. Yes, but not recently 

8. Did you download materials from COL website? 

8.1. materials were provided at meeting 
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8.2. no, hard copies 

8.3. yes 

8.a. 1 . no, prefer hard copy [is it best to circulate materials via COL?] 

8.a.2. yes 

8.b. 1 . getting locked out for password [trouble obtaining study group materials from COL] 

8.b.2. no 

8.C. 1 . online system [if not downloaded, bow were materials provided] 

9. Did you attend webinars re: study groups? 

9.1. No 

10. Did you interact with others via COL website 

10.1. Yes 

10.2. No 

lO.a. 1 . Time constraints [Wbat were reasons for not participating] 

lO.a.2. Too many other things to do 

If. What Web resources did you find helpful 

11.1. create guidelines for high school only 

1 1 .2. implementing ENGAGE/ENGAGE Teacher Edition (formerly known as SRI/BMS) suggestion 

11.3. blog/skype 

1 1 .4. Google docs 

1 1 .5. having a link on the district’s homepage 

11.6. reports available online 

1 1 .7. understanding expected outcomes/expectations 

11.8. assigning each teacher a part of the study 

1 1 .9. loved teaming SG (quote) 

11.10. something to disaggregate scores 

11.11. time 

11.12. getting materials earlier 

f2. What was your impression of SG materials 

12.a.l. 

12.b.l. 

12. C.1. 

13. Overall impression of SG sessions 

13.1. positive feedback 

13.2. negative feedback 

1 3 .a. 1 . have a retreat to learn them 

13. a.2. high school book 

13.a.3. improve facilitator training 
13.a.4. individualize it 

13.a.5. simplify 

1 3 .b . 1 . better understanding of purpose 
13.b.2. yes 

14. What additional study group topics would you like to see offered? 

14.1. career guidance 

14.2. classroom management 

14.3. involving the Community 

14.4. motivation behind behaviors 

14.5. teacher collaboration 

14.6. the ACT test 

14.7. transitions 
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15. How did SG influence you? 

15.1. accountability for other students in the school 

15.2. inspired to improve 

15.3. understanding R&R better 

1 5 .4. influenced way of thinking 

15.5. spending more time talking with teachers 

15.C.1. easy reference for what students need [participation improved your skills - EXPLORE, ENGAGE] 

15.C.2. reference of reasons behind things 
15. c. 3. yes 

15.d. 1. yes, when we were able to implement 

15.d.2. provided focus point [participation improved your skills - career expl. & planning] 

15.d.3. helped better assess need 

16. Have you used or implemented material in SGs 

16.1. no, not enough time 

1 6.2. used some, would like to use more 

16.3. aware of student needs 

16.4. guides conversation 

16.5. yes, implementing action plans 

17. To what extent did you have access to program 

17.1. good access 

18. Problems or harriers that kept you from applying what you learned? 

18.1. no 

18.2. time 

18.3. yes, time and newness 

19. To what extent were administrators committed to making school improvements 

19.1. district should commit further by interviewing earlier 

19.2. example of fully committed administration 

19.3. example of lower scores 

19.4. to a high extent 

19.5. yes 

19.6. half and half 

19.7. yes- with example 

19.8. yes- slowly 

20. Do you think implementation of the info has, or will result in improved student learning 

20.1. got off school improvement list! 

20.2. comments about future implementation 

20.3. yes 

21. Anything else about SG we haven’t discussed 

21.1. no 

2 1 .2. create SG hinder 

21.3. too much information available 

22. Thoughts about teacher PD in general? 

22.1. PD is needed 

22.2. some PD useful, some not 

23. Suggestions about improving professional development in your district 

23.1. make it mandatory 

23.2. let people who want PD go to it 

23.3. make it targeted to individuals 

23.4. make it teacher specific 

23.5. relevance with example 

24. What sort of PD or PD topics do you think are most needed in your school 
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24. 1 . student engagement 

24.2. classroom management 

24.3. diversity training 

24.4. ELL training 

24.5. technology 


Non-Teacher Codes 


1. SchedulingProcess 

1.1. district determined 

1.2. posted on COL 

1.3. emailed TRs 

1.4. failed to include TRs 

2. SchedulingEase 

2.1. easy 

2.2. somewhat difficult 

2.3. very difficult 

3. Schedulingimpediments 

3.1. lack of subs/coverage 

3.2. lack of common schedules 

3.3. lack of communication 

3.4. times offered 

3.5. other 

4. SchedulingBiggestObstacle 

4. 1 . lack of subs 

4.2. lack of communication 

5. SchedulingDifferences 

5.1. f2f scheduled in advance 

5.2. web more to coordinate 

5.3. web unfamiliar 

5.4. web registration/attendance difficult to track 

6. Location 

6.1. classroom/office 

6.2. computer lab/training facility 

6.3. off site location 

6.4. varied 

7. Attendance 

7. 1 . most enthusiastic 

7.2. enthusiasm low 

7.3. enthusiasm/reactions mixed 

7.4. enthusiasm improved 

7.5. enthusiasm higher for f2f 

7.6. district structure impeded 

7.7. technical issues impeded 

7.8. attendance good 

7.9. attendance poor 

7. 10. TRs often late/left early 

7.11. can’t judge 

7.12. lacked incentive 

8. AttendancePressures 

8.1. lack knowledge/expectation 

8.2. lack time/don’t want to be away 

8.3. lack common time 

8.4. uncertainty/reluctance to change 

8.5. skeptical of future 

8.6. other 

9. Leadership Support 

9. 1 . supportive of initiative 

9.2. failed to communicate 

9.3. skeptical of future 

9.4. leadership lacking 
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9.5. support varied 

10. Training Requirement 



10.1. 

required 


10.2. 

seen as optional 


10.3. 

don’t know 

11. 

District Plans 


11.1. 

not in original plans 


11.2. 

in original plans 


11.3. 

communication plan lacking 


11.4. 

TRs not always aware 

12. 

Training Impact 


12.1. 

had impact 


12.2. 

had little or no impact 


12.3. 

impact varied 


12.4. 

can’t judge 

13. 

Format Impact Differences 


13.1. 

f2f bigger impact 


13.2. 

equal impact 

14. 

Implementation Barriers 


14.1. 

time constraints 


14.2. 

classroom problems 


14.3. 

state testing 


14.4. 

other 

15. 

Student Outcome 


15.1. 

improved learning 


15.2. 

hope for improvement 


15.3. 

can’t judge 

16. 

Study Group Impact 


16.1. 

SGs not used 


16.2. 

had an impact 


16.3. 

impact unknown 


16.4. 

want more SGs 

17. 

Commitment to R&R 


17.1. 

TRs committed 


17.2. 

TRs torn 


17.3. 

TRs skeptical 


17.4. 

leadership onboard but lacking 


17.5. 

leadership skeptical 


17.6. 

leadership lacking 


17.7. 

other 

18. 

Different Formats 


18.1. 

seen as different activities 


18.2. 

other 

19. 

Engagement 


19.1. 

seemed engaged 


19.2. 

not as engaged as hoped 


19.3. 

difficult to judge 


19.4. 

tech problems hindered 


19.5. 

small numbers hindered 


19.6. 

sharing/collaboration good 

20. 

Delivery Format 


20.1. 

instructor needed 


20.2. 

instructor not needed 


20.3. 

instructor dependent on content 

21. 

Web 

vF2F 


21.1. 

web has potential 


21.2. 

web offers savings 


21.3. 

web offers flexibility 


21.4. 

web offers safe/familiar environment 


21.5. 

f2f built into district plan 


21.6. 

f2f more familiar to TRs 


21.7. 

other 
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21.8. can go back and review web 

22. Technology Impressions 

22. 1 . technology available was adequate 

22.2. IT setup lacking 

22.3. room configuration/phoneline issue 

22.4. headset/hearing issue 

22.5. connectivity/login issue 

22.6. technology slowed start 

22.7. skills/comfort lacking 

22.8. skills/comfort improved over time 

22.9. other 

23. Web Improvements 

23 . 1 . accountability needed 

23.2. increase flexibility 

23.3. schedule in advance 

23.4. provide clear expectations 

23.5. train on environment/involve IT 

23.6. offer blended solution 

23.7. redesign training for web 

23.8. offer regionally/nationally 

23.9. other 

R1 . schedules or coordinated web-based PD 
R2. followed up with teachers after training 
R3. delivered or observed web-based PD 
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